Bug 23906 - Large emails throw OutOfMemoryError

["Noel J. Bergman" <no...@devtech.com>]

> If I recall my earlier experiments correctly, the MimeMessage(Session,
> InputStream) constructor is the culprit here.

Almost always.

> In my mailserver development I get around this by not constructing a
> MimeMessage (can't see that I need one), writing the message body
> right into a file.

James does that, too.  I think that there are two places that are otherwise,
and I need to fix them.  But please, if you or anyone else has some patches,
please feel free to contribute them.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

RE: Bug 23906 - Large emails throw OutOfMemoryError

["Noel J. Bergman" <no...@devtech.com>]

Some good questions.

> If a database is used as a repository this problem is only prolonged,
cause
> storing or retrieving mails to or from the the database currently loads
the
> entire Mail into a ByteArrayStream.

> Actually im wondering why JDBCMailRepository.store(MailImpl mc) isnt using
> the InputStream of MimeMessageInputStreamSource

There is no question that the James code surrounding MimeMessage, including
MimMessageWrapper and the repositories can be significantly improved.  Feel
free to submit patches.  Please test first.

> JDBCMailRepository.retrieve(String key) loads the Mail with
> ResultSet.getBytes(1) in MimeMessageJDBCSource instead of
> ResultSet.getBinaryStream(1).

To leave it as a stream would requiring leaving the JDBC connection live
until the stream would be closed.  I don't disagree with the possibility,
but we do need to make sure that we are well behaved with respect to the
database server.

The dbfile protocol gets around all of this by storing the Mail instance and
message headers in the database, and the message body in the file system.

	--- Noel


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Bug 23906 - Large emails throw OutOfMemoryError

[Mark Daring <co...@chello.at>]

Richard,

the code you supplied does pretty much the same as MimeMessageWrapper is
doing.
If a database is used as a repository this problem is only prolonged, cause
storing or retrieving mails to or from the the database currently loads the
entire Mail into a ByteArrayStream.
Actually im wondering why JDBCMailRepository.store(MailImpl mc) isnt using
the InputStream of MimeMessageInputStreamSource and
JDBCMailRepository.retrieve(String key) loads the Mail with
ResultSet.getBytes(1) in MimeMessageJDBCSource instead of
ResultSet.getBinaryStream(1).
Maybe someone can explain this?

M


---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org

Re: Bug 23906 - Large emails throw OutOfMemoryError

["Richard O. Hammer" <RO...@EarthLink.net>]

I wrote:
>>In my mailserver development I get around this by not constructing a
>>MimeMessage (can't see that I need one), writing the message body
>>right into a file.

Noel J. Bergman wrote:
> ... if you or anyone else has some patches,
> please feel free to contribute them.

Noel,

Below you can see the little part of my present code that copies in a 
message.

But this is not a "patch", in that I have not considered how it might 
fit into James.  Many of my ideas come from James.  But there are many 
differences too.  For instance, I've decided that I can do what I need 
without ever constructing a MimeMessage out of a message which comes 
into my server.

Rich Hammer
Hillsborough, N.C.
mailscreen.net  -- a hope, a dream


public class Message {
   InternetHeaders headers;
   MailAddress envelopeFrom;
   ArrayList envelopeTo;
   File dataFile; //will hold message data but not headers
   String sentHereFromHost, sentHereFromIP, uniqueID;
   long receivedTime;
   long dataLength = -8,
        totalSize = -22;

   /* A constructor to be used by the SMTP receiver, after envelope 
information
    * has been received, headers have been parsed and updated, and the 
message
    * input stream is positioned to start reading
    * the message body.
    */
   public Message(String remoteHost, String remoteIP, MailAddress 
envelopeFrom,
       ArrayList envelopeTo, InternetHeaders headers, InputStream dataIn)
       throws IOException{
     sentHereFromHost = remoteHost;
     sentHereFromIP = remoteIP;
     this.envelopeFrom = envelopeFrom;
     sortByDomain(envelopeTo);
     this.envelopeTo = envelopeTo;
     this.headers = headers;
     setText(dataIn);
   }

   protected void setText(InputStream dataIn) throws IOException{
     uniqueID = getNewUniqueId();
     dataFile = new File(messagesDirectory, uniqueID);
     OutputStream out = new BufferedOutputStream(new 
FileOutputStream(dataFile));
     byte[] block = new byte[fileTransferBlockSize];
     int readCount;
     while ((readCount = dataIn.read(block)) > -1) {
         out.write(block, 0, readCount);
     }
     out.close();
     dataLength = dataFile.length();
     receivedTime = System.currentTimeMillis();
   }

...

}//class Message

The above code is copyrighted, by which I mean to copy it is right!



---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org