You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Eric Charles <er...@u-mangate.com> on 2010/04/01 09:16:36 UTC
Re: OutOfMemory exception
Hi Norman,
Tks for the smpt-source pointer.
I looked at it, but finally, I think I will develop those little classes
to have more control on it.
I will put it on github.
I launched latest trunk james yesterday evening.
During the night, 3 hours after the launch, the following occured:
KahDB slower and slower...
INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: cleanup took 2330
INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
parsed.;
nested exception is:
org.apache.james.imap.mailbox.StorageException: failed. Transaction
commit failed.;
nested exception is:
<openjpa-1.2.1-r752877:753278 fatal store error>
org.apache.openjpa.persistence.RollbackException: The transaction has
been rolled back. See the nested exceptions for details on the errors
that occurred.
at
org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
at
org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
... 55 more
Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
1363215207 INSERT INTO Message (id, bodyStartOctet, content,
contentOctets, mediaType, subType, textu
alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int)
1260, (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html,
(long) 876]} [code=0, state=XJ001]
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
at
org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
at
org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
So still a OOM exception that was shown by yet-another-component (in
this case, the StoreMailbox).
There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
job).
All 4 files were 0 bytes.
I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
max memory).
With the previous parameters ( -Xmx512m), the process was taking the
whole 512MB.
Tks,
Eric
On 03/31/2010 08:26 PM, Norman Maurer wrote:
> Hi eric,
>
> thx for all your help, maybe smtp-source ( which is included in
> postfix ) can help you with stress testing.
>
> Bye
> Norman
>
> 2010/3/31, Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>>
>> There are not so much .m64 files, so even it it shows that a mail
>> processing didn't reach the end and may have cause a leak, I don't think
>> it would make crash my process so quick (between 10 and 24 hours).
>>
>> I quickly analysed a few dumps last week : there is always a class that
>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>> (don't remember the details).
>> Also notable, the stack trace (the place where the OOM gives problems)
>> varies.
>> Today stack was:
>> org.apache.camel.RuntimeCamelException:
>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>> responseRequired = true, messageId =
>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>> originalTransactionId = null, producerId =
>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>> persistent = true, type = null, priority = 4, groupID = null,
>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>> 0, size = 10718, properties = null, readOnlyProperties = true,
>> readOnlyBody = true, droppable = false}]. Caused by:
>> [java.lang.OutOfMemoryError - Java heap space]
>> at
>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>> at
>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>> ...
>>
>>
>>
>> I also think the tmp files are not the main cause.
>>
>> I have now james running in eclipse and I made already made a little
>> trip in the code. Quite impressive since last time I looked at it.
>> I will try to stress james with a small smtp/pop3 client (I used
>> postage, but a simple class with commons-net behind) may be easier and
>> see what happens (eventually with eclipse profiler).
>>
>> It will be for this weekend for me.
>>
>> Bye,
>>
>> Eric
>>
>>
>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>
>>> Hi Eric,
>>>
>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>> files are still in the tmp folder sometimes.. But I suspect the files
>>> are not the cause of the OOM.
>>>
>>> I didn't get a OOM since yesterday morning (the time I deployed
>>> current trunk version), but just found to new .m64 files..
>>>
>>> So I'm still searching for the real cause. If nothing helps I will
>>> need to use a profiler to find th leak.
>>>
>>> Bye,
>>> Norman
>>>
>>> 2010/3/31 Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hi Norman,
>>>>
>>>> I had defined the Null mailet
>>>>
>>>> <mailet match="HostIsLocal" class="Null">
>>>> <processor> local-address-error</processor>
>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>> </mailet>
>>>>
>>>> but now, I use the standard config
>>>>
>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>> <processor> local-address-error</processor>
>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>> </mailet>
>>>>
>>>> I still have the OOM.
>>>>
>>>> I will now deploy a fresh svn with your last commits and enable the
>>>> ValidRcptHandler.
>>>> I keep you posted with the result,
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> you said all the files are related to address-errors , could you show
>>>>> me your address error processor config?
>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>> the smtpserver.xml file?
>>>>>
>>>>> Thx
>>>>> Norman
>>>>>
>>>>> 2010/3/30, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>> file, on the second after, it is no more there, so the dispose works
>>>>>>> as it should).
>>>>>>>
>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>
>>>>>>> Tks,
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so for
>>>>>>>> <un...@known.com>).
>>>>>>>> They are from various size (with and without attachment).
>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>> thunderbird)
>>>>>>>>
>>>>>>>> However, when I sent a mail to unknown@known.com, I don't see it in
>>>>>>>> the /tmp
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>> could
>>>>>>>>> you check somehow if the .m64 files are files which are related to
>>>>>>>>> successfully delivered mail ?
>>>>>>>>>
>>>>>>>>> Thx,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Norman,
>>>>>>>>>>
>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>
>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>
>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump and
>>>>>>>>>> analyse
>>>>>>>>>> it.
>>>>>>>>>> At the beginning, everything seems ok, and often, when I come back,
>>>>>>>>>> the OOM
>>>>>>>>>> has already occured.
>>>>>>>>>>
>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception (if
>>>>>>>>>> any,
>>>>>>>>>> let's hope not).
>>>>>>>>>>
>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed in
>>>>>>>>>> all
>>>>>>>>>> circumstances.
>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>> files).
>>>>>>>>>> We should also ensure after that there are no other causes to the
>>>>>>>>>> leaks.
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully
>>>>>>>>>>> this
>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2010/3/29 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>> (The
>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>> thats
>>>>>>>>>>>> a
>>>>>>>>>>>> good pointer. I will try to debug the problem later or tomorrow.
>>>>>>>>>>>> But I
>>>>>>>>>>>> suspect you are right about the tmp files and jms producers.. Did
>>>>>>>>>>>> you
>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and
>>>>>>>>>>>>> always
>>>>>>>>>>>>> found
>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but nothing
>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by
>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession, sometimes
>>>>>>>>>>>>> by a
>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
I was refering to expunge while using imap...
Bye,
Norman
2010/4/6 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> Strange, POP3 is now working correctly.
> Regarding expunge, It is disabled by default on thunderbird.
> If I enable it, it seems to remove all mails from the folder, even if I
> don't delete them.
> Remembering the qmail/courrier/... server I was running a few years ago, I
> had a transparent behaviour between pop/imap.
> But I have to take more time to make some tests.
>
> Tks,
> Eric
>
> On 04/06/2010 07:09 PM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> I could only of one change which could maybe improve things and so you
>> don't see the OOM anymore. The patch introduced in revision r930727.
>> This patch changed the behavoir of pop3servers RETR and TOP handler to
>> not load the email in the memory while return it to the client.
>>
>> Do you know which revision you used before ?
>>
>>
>> Some more info would be interesting. Could you tell me after which
>> POP3 command the Mailformed Packets appeared ?
>>
>>
>> Thx,
>> Norman
>>
>> 2010/4/6 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5
>>> day
>>> in prod.
>>>
>>> jmap -dump:file=... (this morning and right now) shows
>>> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the
>>> rest.
>>>
>>> However, I had to redefine all accounts as IMAP in thunderbird to be able
>>> to
>>> read mails:
>>> - the pop3 was working fine, and at a time-being, it hanged.
>>> - Since there, all pop3 requests hang (thunderbird gives me a timeout
>>> after
>>> 2 minutes), simply saying "downloading message 1 of ..." (nothing in the
>>> log
>>> files)
>>> - wireshark shows a correct conversation at the beginning, and afterwards
>>> many "Malformed Packets".
>>> Maybe I could restart james, but I prefer let him work to have a better
>>> view
>>> on the OOM.
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>> On 04/02/2010 09:04 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> I forgot to mention that one cause of the OOM could be the current
>>>> usage of InMemoryMail. At the moment James load the whole MimeMessage
>>>> into memory while processing the camel routes. This is really a bad
>>>> practice in terms of performance and could maybe cause a OOM if you
>>>> processing many big messages which are bigger (the sumary) as the
>>>> avaible memory.
>>>> Could this be the problem on your side ?
>>>>
>>>> I'm currently workin on storing the MimeMessage on an external storage
>>>> and just stream in on demand.. So only the "meta-data" is stored in
>>>> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
>>>> here. But I need a bit more time to see if its the way to go..
>>>>
>>>> I'm also evolating the use of BlobMessages with ActiveMQ.
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>> 2010/4/2 Norman Maurer<no...@apache.org>:
>>>>
>>>>
>>>>>
>>>>> Hi Eric
>>>>>
>>>>> so at least the files not leak anymore, unfortunaly this was the less
>>>>> critical problem.
>>>>> About the errors in pop3server I will have a look and see what could
>>>>> cause this. I'm 100 % sure it's not related to the OOM.
>>>>>
>>>>> Any more infos related to the OOM are welcome.
>>>>>
>>>>> Bye
>>>>> Norman
>>>>>
>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>>
>>>>>> nop. OutOfMemoryError is back.
>>>>>>
>>>>>> I will double check my server, jdk and environment config, but other
>>>>>> java processes are running on the same machine without problem.
>>>>>> I will also have a stress scenario on my local dev PC that runs fine
>>>>>> and
>>>>>> after, I will rerun it on my production server.
>>>>>> Dump analysis and full profiling are also to do.
>>>>>> Some stuff are difficult to simulate : See for example (occurs
>>>>>> regulary)
>>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>>
>>>>>> There are no more .m64 file in /tmp
>>>>>>
>>>>>> Tks,
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>>>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>>>>> 68.62.205.100 for [ric.charles@u-mangate.org,
>>>>>> ric.charles@u-mangate.com]
>>>>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0
>>>>>> ms
>>>>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>>>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>>>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>>>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>>>>> [ric.charles@u-mangate.be]
>>>>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>>>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>>>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0
>>>>>> ms
>>>>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: cleanup took 7274
>>>>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: cleanup took 1141
>>>>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: cleanup took 58100
>>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase
>>>>>> |
>>>>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>>>>> 52100 ms
>>>>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>>>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>>>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>>>>> WARN 16:36:27,919 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>>> space
>>>>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> WARN 16:44:23,231 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.spam'
>>>>>> - trying to recover. Cause: Java heap space
>>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught
>>>>>> an
>>>>>> exception trying to create a JMSException for
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>>>>> valid Object but class java.lang.String
>>>>>> WARN 17:09:11,055 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>>> space
>>>>>> ERROR 17:09:35,017 |
>>>>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>>>>>
>>>>>>
>>>>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>>>>> could not poll endpoint:
>>>>>>
>>>>>>
>>>>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>>>>> caused by: Java heap space
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>> WARN 17:09:55,361 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.spam'
>>>>>> - trying to recover. Cause: Java heap space
>>>>>> WARN 17:09:57,557 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.root'
>>>>>> - trying to recover. Cause: Java heap space
>>>>>> WARN 17:11:00,695 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>>> space
>>>>>> WARN 17:22:19,706 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>>>>> WARN 17:22:42,422 |
>>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>>> Setup
>>>>>> of JMS message listener invoker failed for destination
>>>>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>>>>
>>>>>>
>>>>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> this sounds promising.... Looking forward for your next report.
>>>>>>>
>>>>>>> Bye
>>>>>>> Norman
>>>>>>>
>>>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> James survived the night.
>>>>>>>> 300MB (exactly the same as when it started).
>>>>>>>> No m64 file in tmp.
>>>>>>>>
>>>>>>>> I will post another update this evening.
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>>>>
>>>>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>>>>
>>>>>>>>> Bye,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello Norman,
>>>>>>>>>> Done :)
>>>>>>>>>> Tks, Eric
>>>>>>>>>>
>>>>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I
>>>>>>>>>> am
>>>>>>>>>> using.
>>>>>>>>>> PS2 : I only checkout
>>>>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>>>>> and
>>>>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>>>>> version
>>>>>>>>>> of
>>>>>>>>>> imap-mailbox, jsieve,... are taken from the central maven
>>>>>>>>>> repository
>>>>>>>>>> and
>>>>>>>>>> I
>>>>>>>>>> don't have to rebuild those projects.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> its me again ;) Could you please svn up to latest trunk and
>>>>>>>>>>> try
>>>>>>>>>>> again
>>>>>>>>>>> ?
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>>>>> background
>>>>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and
>>>>>>>>>>>> 3000
>>>>>>>>>>>> with 5 mb.
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher
>>>>>>>>>>>> then
>>>>>>>>>>>> 700m.
>>>>>>>>>>>>
>>>>>>>>>>>> Bye.
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
Strange, POP3 is now working correctly.
Regarding expunge, It is disabled by default on thunderbird.
If I enable it, it seems to remove all mails from the folder, even if I
don't delete them.
Remembering the qmail/courrier/... server I was running a few years ago,
I had a transparent behaviour between pop/imap.
But I have to take more time to make some tests.
Tks,
Eric
On 04/06/2010 07:09 PM, Norman Maurer wrote:
> Hi Eric,
>
> I could only of one change which could maybe improve things and so you
> don't see the OOM anymore. The patch introduced in revision r930727.
> This patch changed the behavoir of pop3servers RETR and TOP handler to
> not load the email in the memory while return it to the client.
>
> Do you know which revision you used before ?
>
>
> Some more info would be interesting. Could you tell me after which
> POP3 command the Mailformed Packets appeared ?
>
>
> Thx,
> Norman
>
> 2010/4/6 Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>>
>> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day
>> in prod.
>>
>> jmap -dump:file=... (this morning and right now) shows
>> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the
>> rest.
>>
>> However, I had to redefine all accounts as IMAP in thunderbird to be able to
>> read mails:
>> - the pop3 was working fine, and at a time-being, it hanged.
>> - Since there, all pop3 requests hang (thunderbird gives me a timeout after
>> 2 minutes), simply saying "downloading message 1 of ..." (nothing in the log
>> files)
>> - wireshark shows a correct conversation at the beginning, and afterwards
>> many "Malformed Packets".
>> Maybe I could restart james, but I prefer let him work to have a better view
>> on the OOM.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/02/2010 09:04 PM, Norman Maurer wrote:
>>
>>> Hi Eric,
>>>
>>> I forgot to mention that one cause of the OOM could be the current
>>> usage of InMemoryMail. At the moment James load the whole MimeMessage
>>> into memory while processing the camel routes. This is really a bad
>>> practice in terms of performance and could maybe cause a OOM if you
>>> processing many big messages which are bigger (the sumary) as the
>>> avaible memory.
>>> Could this be the problem on your side ?
>>>
>>> I'm currently workin on storing the MimeMessage on an external storage
>>> and just stream in on demand.. So only the "meta-data" is stored in
>>> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
>>> here. But I need a bit more time to see if its the way to go..
>>>
>>> I'm also evolating the use of BlobMessages with ActiveMQ.
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>> 2010/4/2 Norman Maurer<no...@apache.org>:
>>>
>>>
>>>> Hi Eric
>>>>
>>>> so at least the files not leak anymore, unfortunaly this was the less
>>>> critical problem.
>>>> About the errors in pop3server I will have a look and see what could
>>>> cause this. I'm 100 % sure it's not related to the OOM.
>>>>
>>>> Any more infos related to the OOM are welcome.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>> nop. OutOfMemoryError is back.
>>>>>
>>>>> I will double check my server, jdk and environment config, but other
>>>>> java processes are running on the same machine without problem.
>>>>> I will also have a stress scenario on my local dev PC that runs fine and
>>>>> after, I will rerun it on my production server.
>>>>> Dump analysis and full profiling are also to do.
>>>>> Some stuff are difficult to simulate : See for example (occurs regulary)
>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>>
>>>>> There are no more .m64 file in /tmp
>>>>>
>>>>> Tks,
>>>>> Eric
>>>>>
>>>>>
>>>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>>>> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
>>>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>>>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>>>> [ric.charles@u-mangate.be]
>>>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
>>>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 7274
>>>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 1141
>>>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 58100
>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>>>> 52100 ms
>>>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>>>> WARN 16:36:27,919 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> WARN 16:44:23,231 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>>> - trying to recover. Cause: Java heap space
>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>>>>> exception trying to create a JMSException for
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> WARN 17:09:11,055 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> ERROR 17:09:35,017 |
>>>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>>>>
>>>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>>>> could not poll endpoint:
>>>>>
>>>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>>>> caused by: Java heap space
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> WARN 17:09:55,361 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>>> - trying to recover. Cause: Java heap space
>>>>> WARN 17:09:57,557 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination 'processor.root'
>>>>> - trying to recover. Cause: Java heap space
>>>>> WARN 17:11:00,695 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> WARN 17:22:19,706 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>>>> WARN 17:22:42,422 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>>>
>>>>>
>>>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> this sounds promising.... Looking forward for your next report.
>>>>>>
>>>>>> Bye
>>>>>> Norman
>>>>>>
>>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> James survived the night.
>>>>>>> 300MB (exactly the same as when it started).
>>>>>>> No m64 file in tmp.
>>>>>>>
>>>>>>> I will post another update this evening.
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>>>
>>>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>>>
>>>>>>>> Bye,
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hello Norman,
>>>>>>>>> Done :)
>>>>>>>>> Tks, Eric
>>>>>>>>>
>>>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>>>>>>> using.
>>>>>>>>> PS2 : I only checkout
>>>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>>>> and
>>>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>>>> version
>>>>>>>>> of
>>>>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>>>>>> and
>>>>>>>>> I
>>>>>>>>> don't have to rebuild those projects.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>>>>>> again
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> Thx,
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>>>> background
>>>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and
>>>>>>>>>>> 3000
>>>>>>>>>>> with 5 mb.
>>>>>>>>>>>
>>>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>>>>>> 700m.
>>>>>>>>>>>
>>>>>>>>>>> Bye.
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
I could only of one change which could maybe improve things and so you
don't see the OOM anymore. The patch introduced in revision r930727.
This patch changed the behavoir of pop3servers RETR and TOP handler to
not load the email in the memory while return it to the client.
Do you know which revision you used before ?
Some more info would be interesting. Could you tell me after which
POP3 command the Mailformed Packets appeared ?
Thx,
Norman
2010/4/6 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5 day
> in prod.
>
> jmap -dump:file=... (this morning and right now) shows
> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the
> rest.
>
> However, I had to redefine all accounts as IMAP in thunderbird to be able to
> read mails:
> - the pop3 was working fine, and at a time-being, it hanged.
> - Since there, all pop3 requests hang (thunderbird gives me a timeout after
> 2 minutes), simply saying "downloading message 1 of ..." (nothing in the log
> files)
> - wireshark shows a correct conversation at the beginning, and afterwards
> many "Malformed Packets".
> Maybe I could restart james, but I prefer let him work to have a better view
> on the OOM.
>
> Tks,
>
> Eric
>
>
> On 04/02/2010 09:04 PM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> I forgot to mention that one cause of the OOM could be the current
>> usage of InMemoryMail. At the moment James load the whole MimeMessage
>> into memory while processing the camel routes. This is really a bad
>> practice in terms of performance and could maybe cause a OOM if you
>> processing many big messages which are bigger (the sumary) as the
>> avaible memory.
>> Could this be the problem on your side ?
>>
>> I'm currently workin on storing the MimeMessage on an external storage
>> and just stream in on demand.. So only the "meta-data" is stored in
>> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
>> here. But I need a bit more time to see if its the way to go..
>>
>> I'm also evolating the use of BlobMessages with ActiveMQ.
>>
>> Bye,
>> Norman
>>
>>
>> 2010/4/2 Norman Maurer<no...@apache.org>:
>>
>>>
>>> Hi Eric
>>>
>>> so at least the files not leak anymore, unfortunaly this was the less
>>> critical problem.
>>> About the errors in pop3server I will have a look and see what could
>>> cause this. I'm 100 % sure it's not related to the OOM.
>>>
>>> Any more infos related to the OOM are welcome.
>>>
>>> Bye
>>> Norman
>>>
>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>
>>>>
>>>> nop. OutOfMemoryError is back.
>>>>
>>>> I will double check my server, jdk and environment config, but other
>>>> java processes are running on the same machine without problem.
>>>> I will also have a stress scenario on my local dev PC that runs fine and
>>>> after, I will rerun it on my production server.
>>>> Dump analysis and full profiling are also to do.
>>>> Some stuff are difficult to simulate : See for example (occurs regulary)
>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>>
>>>> There are no more .m64 file in /tmp
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>>> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
>>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>>> [ric.charles@u-mangate.be]
>>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
>>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 7274
>>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 1141
>>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 58100
>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>>> 52100 ms
>>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>>> WARN 16:36:27,919 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>> space
>>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> WARN 16:44:23,231 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>> - trying to recover. Cause: Java heap space
>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>>>> exception trying to create a JMSException for
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> WARN 17:09:11,055 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>> space
>>>> ERROR 17:09:35,017 |
>>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>>>
>>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>>> could not poll endpoint:
>>>>
>>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>>> caused by: Java heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> WARN 17:09:55,361 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>> - trying to recover. Cause: Java heap space
>>>> WARN 17:09:57,557 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination 'processor.root'
>>>> - trying to recover. Cause: Java heap space
>>>> WARN 17:11:00,695 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>> space
>>>> WARN 17:22:19,706 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>>> WARN 17:22:42,422 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>>
>>>>
>>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>>
>>>>>
>>>>> Hi Eric,
>>>>>
>>>>> this sounds promising.... Looking forward for your next report.
>>>>>
>>>>> Bye
>>>>> Norman
>>>>>
>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>>
>>>>>> James survived the night.
>>>>>> 300MB (exactly the same as when it started).
>>>>>> No m64 file in tmp.
>>>>>>
>>>>>> I will post another update this evening.
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>>
>>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hello Norman,
>>>>>>>> Done :)
>>>>>>>> Tks, Eric
>>>>>>>>
>>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>>>>>> using.
>>>>>>>> PS2 : I only checkout
>>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>>> and
>>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>>> version
>>>>>>>> of
>>>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>>>>> and
>>>>>>>> I
>>>>>>>> don't have to rebuild those projects.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>>>>> again
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> Thx,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>>> background
>>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and
>>>>>>>>>> 3000
>>>>>>>>>> with 5 mb.
>>>>>>>>>>
>>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>>>>> 700m.
>>>>>>>>>>
>>>>>>>>>> Bye.
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Nice to hear this..
Did you switch to IMAP netty yet ?
Bye,
Norman
2010/4/20 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> It is now working as I would expect: deleted mails via imap are no more
> dowloaded via pop3, after thunderbird exit of course doing the expunge.
>
> Just for the record, I am pretty sure the behaviour was not correct with the
> versions that were producin oom.
> Probably nothing to do with each others; your commits seem to have solved
> much issues.
>
> Tks,
>
> Eric
>
>
> On 01/-10/-28163 08:59 PM, Norman Maurer wrote:
>>
>> Hmm.. are you sure thunderbird is calling expunge ?
>>
>> Thx,
>> Norman
>>
>> 2010/4/6 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> One more information.
>>> Actions taken via IMAP are net reflected on POP3 : deleting a mail via
>>> IMAP,
>>> you will still have to download it via POP3.
>>> I don't know if this can help, neither if the goal is to have the same
>>> behaviour via IMAP and POP3.
>>> Tks,
>>> Eric
>>>
>>> On 04/06/2010 05:41 PM, Eric Charles wrote:
>>>
>>>>
>>>> Hi Norman,
>>>>
>>>> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5
>>>> day in prod.
>>>>
>>>> jmap -dump:file=... (this morning and right now) shows
>>>> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for
>>>> the
>>>> rest.
>>>>
>>>> However, I had to redefine all accounts as IMAP in thunderbird to be
>>>> able
>>>> to read mails:
>>>> - the pop3 was working fine, and at a time-being, it hanged.
>>>> - Since there, all pop3 requests hang (thunderbird gives me a timeout
>>>> after 2 minutes), simply saying "downloading message 1 of ..." (nothing
>>>> in
>>>> the log files)
>>>> - wireshark shows a correct conversation at the beginning, and
>>>> afterwards
>>>> many "Malformed Packets".
>>>> Maybe I could restart james, but I prefer let him work to have a better
>>>> view on the OOM.
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
It is now working as I would expect: deleted mails via imap are no more
dowloaded via pop3, after thunderbird exit of course doing the expunge.
Just for the record, I am pretty sure the behaviour was not correct with
the versions that were producin oom.
Probably nothing to do with each others; your commits seem to have
solved much issues.
Tks,
Eric
On 01/-10/-28163 08:59 PM, Norman Maurer wrote:
> Hmm.. are you sure thunderbird is calling expunge ?
>
> Thx,
> Norman
>
> 2010/4/6 Eric Charles<er...@u-mangate.com>:
>
>> One more information.
>> Actions taken via IMAP are net reflected on POP3 : deleting a mail via IMAP,
>> you will still have to download it via POP3.
>> I don't know if this can help, neither if the goal is to have the same
>> behaviour via IMAP and POP3.
>> Tks,
>> Eric
>>
>> On 04/06/2010 05:41 PM, Eric Charles wrote:
>>
>>> Hi Norman,
>>>
>>> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5
>>> day in prod.
>>>
>>> jmap -dump:file=... (this morning and right now) shows
>>> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the
>>> rest.
>>>
>>> However, I had to redefine all accounts as IMAP in thunderbird to be able
>>> to read mails:
>>> - the pop3 was working fine, and at a time-being, it hanged.
>>> - Since there, all pop3 requests hang (thunderbird gives me a timeout
>>> after 2 minutes), simply saying "downloading message 1 of ..." (nothing in
>>> the log files)
>>> - wireshark shows a correct conversation at the beginning, and afterwards
>>> many "Malformed Packets".
>>> Maybe I could restart james, but I prefer let him work to have a better
>>> view on the OOM.
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hmm.. are you sure thunderbird is calling expunge ?
Thx,
Norman
2010/4/6 Eric Charles <er...@u-mangate.com>:
> One more information.
> Actions taken via IMAP are net reflected on POP3 : deleting a mail via IMAP,
> you will still have to download it via POP3.
> I don't know if this can help, neither if the goal is to have the same
> behaviour via IMAP and POP3.
> Tks,
> Eric
>
> On 04/06/2010 05:41 PM, Eric Charles wrote:
>>
>> Hi Norman,
>>
>> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5
>> day in prod.
>>
>> jmap -dump:file=... (this morning and right now) shows
>> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for the
>> rest.
>>
>> However, I had to redefine all accounts as IMAP in thunderbird to be able
>> to read mails:
>> - the pop3 was working fine, and at a time-being, it hanged.
>> - Since there, all pop3 requests hang (thunderbird gives me a timeout
>> after 2 minutes), simply saying "downloading message 1 of ..." (nothing in
>> the log files)
>> - wireshark shows a correct conversation at the beginning, and afterwards
>> many "Malformed Packets".
>> Maybe I could restart james, but I prefer let him work to have a better
>> view on the OOM.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/02/2010 09:04 PM, Norman Maurer wrote:
>>>
>>> Hi Eric,
>>>
>>> I forgot to mention that one cause of the OOM could be the current
>>> usage of InMemoryMail. At the moment James load the whole MimeMessage
>>> into memory while processing the camel routes. This is really a bad
>>> practice in terms of performance and could maybe cause a OOM if you
>>> processing many big messages which are bigger (the sumary) as the
>>> avaible memory.
>>> Could this be the problem on your side ?
>>>
>>> I'm currently workin on storing the MimeMessage on an external storage
>>> and just stream in on demand.. So only the "meta-data" is stored in
>>> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
>>> here. But I need a bit more time to see if its the way to go..
>>>
>>> I'm also evolating the use of BlobMessages with ActiveMQ.
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>> 2010/4/2 Norman Maurer<no...@apache.org>:
>>>>
>>>> Hi Eric
>>>>
>>>> so at least the files not leak anymore, unfortunaly this was the less
>>>> critical problem.
>>>> About the errors in pop3server I will have a look and see what could
>>>> cause this. I'm 100 % sure it's not related to the OOM.
>>>>
>>>> Any more infos related to the OOM are welcome.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>> nop. OutOfMemoryError is back.
>>>>>
>>>>> I will double check my server, jdk and environment config, but other
>>>>> java processes are running on the same machine without problem.
>>>>> I will also have a stress scenario on my local dev PC that runs fine
>>>>> and
>>>>> after, I will rerun it on my production server.
>>>>> Dump analysis and full profiling are also to do.
>>>>> Some stuff are difficult to simulate : See for example (occurs
>>>>> regulary)
>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>>
>>>>> There are no more .m64 file in /tmp
>>>>>
>>>>> Tks,
>>>>> Eric
>>>>>
>>>>>
>>>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>>>> 68.62.205.100 for [ric.charles@u-mangate.org,
>>>>> ric.charles@u-mangate.com]
>>>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>>>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>>>> [ric.charles@u-mangate.be]
>>>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0
>>>>> ms
>>>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 7274
>>>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 1141
>>>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: cleanup took 58100
>>>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>>>> 52100 ms
>>>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>>>> WARN 16:36:27,919 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> WARN 16:44:23,231 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>>> - trying to recover. Cause: Java heap space
>>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>>>>> exception trying to create a JMSException for
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>>>> valid Object but class java.lang.String
>>>>> WARN 17:09:11,055 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> ERROR 17:09:35,017 |
>>>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>>>>
>>>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>>>> could not poll endpoint:
>>>>>
>>>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>>>> caused by: Java heap space
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>> WARN 17:09:55,361 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination 'processor.spam'
>>>>> - trying to recover. Cause: Java heap space
>>>>> WARN 17:09:57,557 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination 'processor.root'
>>>>> - trying to recover. Cause: Java heap space
>>>>> WARN 17:11:00,695 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.local-address-error' - trying to recover. Cause: Java heap
>>>>> space
>>>>> WARN 17:22:19,706 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>>>> WARN 17:22:42,422 |
>>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>>> Setup
>>>>> of JMS message listener invoker failed for destination
>>>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>>>
>>>>>
>>>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> this sounds promising.... Looking forward for your next report.
>>>>>>
>>>>>> Bye
>>>>>> Norman
>>>>>>
>>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>> James survived the night.
>>>>>>> 300MB (exactly the same as when it started).
>>>>>>> No m64 file in tmp.
>>>>>>>
>>>>>>> I will post another update this evening.
>>>>>>>
>>>>>>> Tks,
>>>>>>>
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>>>
>>>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>>>
>>>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>>>
>>>>>>>> Bye,
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hello Norman,
>>>>>>>>> Done :)
>>>>>>>>> Tks, Eric
>>>>>>>>>
>>>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I
>>>>>>>>> am
>>>>>>>>> using.
>>>>>>>>> PS2 : I only checkout
>>>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>>>> and
>>>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>>>> version
>>>>>>>>> of
>>>>>>>>> imap-mailbox, jsieve,... are taken from the central maven
>>>>>>>>> repository
>>>>>>>>> and
>>>>>>>>> I
>>>>>>>>> don't have to rebuild those projects.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>>>>>> again
>>>>>>>>>> ?
>>>>>>>>>>
>>>>>>>>>> Thx,
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>>>> background
>>>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and
>>>>>>>>>>> 3000
>>>>>>>>>>> with 5 mb.
>>>>>>>>>>>
>>>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>>>>>> 700m.
>>>>>>>>>>>
>>>>>>>>>>> Bye.
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
One more information.
Actions taken via IMAP are net reflected on POP3 : deleting a mail via
IMAP, you will still have to download it via POP3.
I don't know if this can help, neither if the goal is to have the same
behaviour via IMAP and POP3.
Tks,
Eric
On 04/06/2010 05:41 PM, Eric Charles wrote:
> Hi Norman,
>
> james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since
> 1,5 day in prod.
>
> jmap -dump:file=... (this morning and right now) shows
> org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for
> the rest.
>
> However, I had to redefine all accounts as IMAP in thunderbird to be
> able to read mails:
> - the pop3 was working fine, and at a time-being, it hanged.
> - Since there, all pop3 requests hang (thunderbird gives me a timeout
> after 2 minutes), simply saying "downloading message 1 of ..."
> (nothing in the log files)
> - wireshark shows a correct conversation at the beginning, and
> afterwards many "Malformed Packets".
> Maybe I could restart james, but I prefer let him work to have a
> better view on the OOM.
>
> Tks,
>
> Eric
>
>
> On 04/02/2010 09:04 PM, Norman Maurer wrote:
>> Hi Eric,
>>
>> I forgot to mention that one cause of the OOM could be the current
>> usage of InMemoryMail. At the moment James load the whole MimeMessage
>> into memory while processing the camel routes. This is really a bad
>> practice in terms of performance and could maybe cause a OOM if you
>> processing many big messages which are bigger (the sumary) as the
>> avaible memory.
>> Could this be the problem on your side ?
>>
>> I'm currently workin on storing the MimeMessage on an external storage
>> and just stream in on demand.. So only the "meta-data" is stored in
>> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
>> here. But I need a bit more time to see if its the way to go..
>>
>> I'm also evolating the use of BlobMessages with ActiveMQ.
>>
>> Bye,
>> Norman
>>
>>
>> 2010/4/2 Norman Maurer<no...@apache.org>:
>>> Hi Eric
>>>
>>> so at least the files not leak anymore, unfortunaly this was the less
>>> critical problem.
>>> About the errors in pop3server I will have a look and see what could
>>> cause this. I'm 100 % sure it's not related to the OOM.
>>>
>>> Any more infos related to the OOM are welcome.
>>>
>>> Bye
>>> Norman
>>>
>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>> nop. OutOfMemoryError is back.
>>>>
>>>> I will double check my server, jdk and environment config, but other
>>>> java processes are running on the same machine without problem.
>>>> I will also have a stress scenario on my local dev PC that runs
>>>> fine and
>>>> after, I will rerun it on my production server.
>>>> Dump analysis and full profiling are also to do.
>>>> Some stuff are difficult to simulate : See for example (occurs
>>>> regulary)
>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>>
>>>> There are no more .m64 file in /tmp
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>>> 68.62.205.100 for [ric.charles@u-mangate.org,
>>>> ric.charles@u-mangate.com]
>>>> INFO 16:25:00,097 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 989 ms, Index update took
>>>> 0 ms
>>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>>> [ric.charles@u-mangate.be]
>>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>>> INFO 16:25:09,908 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took
>>>> 0 ms
>>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:25:41,212 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 7274
>>>> INFO 16:25:55,612 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 1141
>>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> INFO 16:27:00,003 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: cleanup took 58100
>>>> INFO 16:27:00,003 |
>>>> org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>>> 52100 ms
>>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>>> WARN 16:36:27,919 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java
>>>> heap space
>>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> WARN 16:44:23,231 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.spam'
>>>> - trying to recover. Cause: Java heap space
>>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection |
>>>> Caught an
>>>> exception trying to create a JMSException for
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>>> valid Object but class java.lang.String
>>>> WARN 17:09:11,055 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java
>>>> heap space
>>>> ERROR 17:09:35,017 |
>>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>>>
>>>> could not poll endpoint:
>>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>>>
>>>> caused by: Java heap space
>>>> java.lang.OutOfMemoryError: Java heap space
>>>> WARN 17:09:55,361 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.spam'
>>>> - trying to recover. Cause: Java heap space
>>>> WARN 17:09:57,557 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.root'
>>>> - trying to recover. Cause: Java heap space
>>>> WARN 17:11:00,695 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.local-address-error' - trying to recover. Cause: Java
>>>> heap space
>>>> WARN 17:22:19,706 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>>> WARN 17:22:42,422 |
>>>> org.springframework.jms.listener.DefaultMessageListenerContainer |
>>>> Setup
>>>> of JMS message listener invoker failed for destination
>>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>>
>>>>
>>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>>> Hi Eric,
>>>>>
>>>>> this sounds promising.... Looking forward for your next report.
>>>>>
>>>>> Bye
>>>>> Norman
>>>>>
>>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>> James survived the night.
>>>>>> 300MB (exactly the same as when it started).
>>>>>> No m64 file in tmp.
>>>>>>
>>>>>> I will post another update this evening.
>>>>>>
>>>>>> Tks,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>>
>>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>>
>>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>> Hello Norman,
>>>>>>>> Done :)
>>>>>>>> Tks, Eric
>>>>>>>>
>>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList
>>>>>>>> I am
>>>>>>>> using.
>>>>>>>> PS2 : I only checkout
>>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>>> and
>>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>>> version
>>>>>>>> of
>>>>>>>> imap-mailbox, jsieve,... are taken from the central maven
>>>>>>>> repository
>>>>>>>> and
>>>>>>>> I
>>>>>>>> don't have to rebuild those projects.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> its me again ;) Could you please svn up to latest trunk
>>>>>>>>> and try
>>>>>>>>> again
>>>>>>>>> ?
>>>>>>>>>
>>>>>>>>> Thx,
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>>> background
>>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it
>>>>>>>>>> and 3000
>>>>>>>>>> with 5 mb.
>>>>>>>>>>
>>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher
>>>>>>>>>> then
>>>>>>>>>> 700m.
>>>>>>>>>>
>>>>>>>>>> Bye.
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>>
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>>
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
Hi Norman,
james revision 930727 (mina RC1, jdk 1.6.0_18) up-and-running since 1,5
day in prod.
jmap -dump:file=... (this morning and right now) shows
org.apache.derby.impl.services.cache.Clock with 15% of Heap, less for
the rest.
However, I had to redefine all accounts as IMAP in thunderbird to be
able to read mails:
- the pop3 was working fine, and at a time-being, it hanged.
- Since there, all pop3 requests hang (thunderbird gives me a timeout
after 2 minutes), simply saying "downloading message 1 of ..." (nothing
in the log files)
- wireshark shows a correct conversation at the beginning, and
afterwards many "Malformed Packets".
Maybe I could restart james, but I prefer let him work to have a better
view on the OOM.
Tks,
Eric
On 04/02/2010 09:04 PM, Norman Maurer wrote:
> Hi Eric,
>
> I forgot to mention that one cause of the OOM could be the current
> usage of InMemoryMail. At the moment James load the whole MimeMessage
> into memory while processing the camel routes. This is really a bad
> practice in terms of performance and could maybe cause a OOM if you
> processing many big messages which are bigger (the sumary) as the
> avaible memory.
> Could this be the problem on your side ?
>
> I'm currently workin on storing the MimeMessage on an external storage
> and just stream in on demand.. So only the "meta-data" is stored in
> jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
> here. But I need a bit more time to see if its the way to go..
>
> I'm also evolating the use of BlobMessages with ActiveMQ.
>
> Bye,
> Norman
>
>
> 2010/4/2 Norman Maurer<no...@apache.org>:
>
>> Hi Eric
>>
>> so at least the files not leak anymore, unfortunaly this was the less
>> critical problem.
>> About the errors in pop3server I will have a look and see what could
>> cause this. I'm 100 % sure it's not related to the OOM.
>>
>> Any more infos related to the OOM are welcome.
>>
>> Bye
>> Norman
>>
>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>
>>> nop. OutOfMemoryError is back.
>>>
>>> I will double check my server, jdk and environment config, but other
>>> java processes are running on the same machine without problem.
>>> I will also have a stress scenario on my local dev PC that runs fine and
>>> after, I will rerun it on my production server.
>>> Dump analysis and full profiling are also to do.
>>> Some stuff are difficult to simulate : See for example (occurs regulary)
>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>>
>>> There are no more .m64 file in /tmp
>>>
>>> Tks,
>>> Eric
>>>
>>>
>>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>>> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
>>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>>> [ric.charles@u-mangate.be]
>>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
>>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 7274
>>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 1141
>>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: cleanup took 58100
>>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>>> 52100 ms
>>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>>> WARN 16:36:27,919 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> WARN 16:44:23,231 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.spam'
>>> - trying to recover. Cause: Java heap space
>>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>>> exception trying to create a JMSException for
>>> java.lang.OutOfMemoryError: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>>> valid Object but class java.lang.String
>>> WARN 17:09:11,055 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> ERROR 17:09:35,017 |
>>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>>> could not poll endpoint:
>>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>>> caused by: Java heap space
>>> java.lang.OutOfMemoryError: Java heap space
>>> WARN 17:09:55,361 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.spam'
>>> - trying to recover. Cause: Java heap space
>>> WARN 17:09:57,557 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination 'processor.root'
>>> - trying to recover. Cause: Java heap space
>>> WARN 17:11:00,695 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>>> WARN 17:22:19,706 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.bounces' - trying to recover. Cause: Java heap space
>>> WARN 17:22:42,422 |
>>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>>> of JMS message listener invoker failed for destination
>>> 'processor.transport' - trying to recover. Cause: Java heap space
>>>
>>>
>>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>>
>>>> Hi Eric,
>>>>
>>>> this sounds promising.... Looking forward for your next report.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>> James survived the night.
>>>>> 300MB (exactly the same as when it started).
>>>>> No m64 file in tmp.
>>>>>
>>>>> I will post another update this evening.
>>>>>
>>>>> Tks,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>> Yeah thats right.. just checout server/trunk.
>>>>>>
>>>>>> I'm looking forward to hear your feedback ;)
>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hello Norman,
>>>>>>> Done :)
>>>>>>> Tks, Eric
>>>>>>>
>>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>>>>> using.
>>>>>>> PS2 : I only checkout
>>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>>> and
>>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>>> version
>>>>>>> of
>>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>>>> and
>>>>>>> I
>>>>>>> don't have to rebuild those projects.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>>>> again
>>>>>>>> ?
>>>>>>>>
>>>>>>>> Thx,
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>>> background
>>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>>>>>>> with 5 mb.
>>>>>>>>>
>>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>>>> 700m.
>>>>>>>>>
>>>>>>>>> Bye.
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@apache.org>.
Hi Eric,
I forgot to mention that one cause of the OOM could be the current
usage of InMemoryMail. At the moment James load the whole MimeMessage
into memory while processing the camel routes. This is really a bad
practice in terms of performance and could maybe cause a OOM if you
processing many big messages which are bigger (the sumary) as the
avaible memory.
Could this be the problem on your side ?
I'm currently workin on storing the MimeMessage on an external storage
and just stream in on demand.. So only the "meta-data" is stored in
jms and the rest in a db / jcr etc.. I have a proof-of-concept workin
here. But I need a bit more time to see if its the way to go..
I'm also evolating the use of BlobMessages with ActiveMQ.
Bye,
Norman
2010/4/2 Norman Maurer <no...@apache.org>:
> Hi Eric
>
> so at least the files not leak anymore, unfortunaly this was the less
> critical problem.
> About the errors in pop3server I will have a look and see what could
> cause this. I'm 100 % sure it's not related to the OOM.
>
> Any more infos related to the OOM are welcome.
>
> Bye
> Norman
>
> 2010/4/2, Eric Charles <er...@u-mangate.com>:
>> nop. OutOfMemoryError is back.
>>
>> I will double check my server, jdk and environment config, but other
>> java processes are running on the same machine without problem.
>> I will also have a stress scenario on my local dev PC that runs fine and
>> after, I will rerun it on my production server.
>> Dump analysis and full profiling are also to do.
>> Some stuff are difficult to simulate : See for example (occurs regulary)
>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>>
>> There are no more .m64 file in /tmp
>>
>> Tks,
>> Eric
>>
>>
>> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
>> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
>> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
>> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
>> INFO 16:25:00,099 | james.mailetcontext | Storing mail
>> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
>> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
>> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
>> [ric.charles@u-mangate.be]
>> INFO 16:25:08,723 | james.mailetcontext | Storing mail
>> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
>> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
>> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: cleanup took 7274
>> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: cleanup took 1141
>> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: cleanup took 58100
>> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
>> Slow KahaDB access: Journal append took: 10032 ms, Index update took
>> 52100 ms
>> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
>> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
>> 213.163.116.78 for [angelo.biddau@u-mangate.net]
>> WARN 16:36:27,919 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination
>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> WARN 16:44:23,231 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination 'processor.spam'
>> - trying to recover. Cause: Java heap space
>> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
>> exception trying to create a JMSException for
>> java.lang.OutOfMemoryError: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
>> valid Object but class java.lang.String
>> WARN 17:09:11,055 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination
>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>> ERROR 17:09:35,017 |
>> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
>> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
>> could not poll endpoint:
>> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
>> caused by: Java heap space
>> java.lang.OutOfMemoryError: Java heap space
>> WARN 17:09:55,361 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination 'processor.spam'
>> - trying to recover. Cause: Java heap space
>> WARN 17:09:57,557 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination 'processor.root'
>> - trying to recover. Cause: Java heap space
>> WARN 17:11:00,695 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination
>> 'processor.local-address-error' - trying to recover. Cause: Java heap space
>> WARN 17:22:19,706 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination
>> 'processor.bounces' - trying to recover. Cause: Java heap space
>> WARN 17:22:42,422 |
>> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
>> of JMS message listener invoker failed for destination
>> 'processor.transport' - trying to recover. Cause: Java heap space
>>
>>
>> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>>> Hi Eric,
>>>
>>> this sounds promising.... Looking forward for your next report.
>>>
>>> Bye
>>> Norman
>>>
>>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>>
>>>> James survived the night.
>>>> 300MB (exactly the same as when it started).
>>>> No m64 file in tmp.
>>>>
>>>> I will post another update this evening.
>>>>
>>>> Tks,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>>
>>>>> Yeah thats right.. just checout server/trunk.
>>>>>
>>>>> I'm looking forward to hear your feedback ;)
>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>>
>>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>> Hello Norman,
>>>>>> Done :)
>>>>>> Tks, Eric
>>>>>>
>>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>>>> using.
>>>>>> PS2 : I only checkout
>>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>>> and
>>>>>> not the other projects. From what I read from the poms, snapshot
>>>>>> version
>>>>>> of
>>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>>> and
>>>>>> I
>>>>>> don't have to rebuild those projects.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>>> again
>>>>>>> ?
>>>>>>>
>>>>>>> Thx,
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>>
>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>>> background
>>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>>>>>> with 5 mb.
>>>>>>>>
>>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>>> 700m.
>>>>>>>>
>>>>>>>> Bye.
>>>>>>>> Norman
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@apache.org>.
Hi Eric
so at least the files not leak anymore, unfortunaly this was the less
critical problem.
About the errors in pop3server I will have a look and see what could
cause this. I'm 100 % sure it's not related to the OOM.
Any more infos related to the OOM are welcome.
Bye
Norman
2010/4/2, Eric Charles <er...@u-mangate.com>:
> nop. OutOfMemoryError is back.
>
> I will double check my server, jdk and environment config, but other
> java processes are running on the same machine without problem.
> I will also have a stress scenario on my local dev PC that runs fine and
> after, I will rerun it on my production server.
> Dump analysis and full profiling are also to do.
> Some stuff are difficult to simulate : See for example (occurs regulary)
> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
>
> There are no more .m64 file in /tmp
>
> Tks,
> Eric
>
>
> ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
> Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
> 68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
> INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
> INFO 16:25:00,099 | james.mailetcontext | Storing mail
> Mail1270218298023-9820-!226071 in file://var/mail/address-error/
> INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
> Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
> [ric.charles@u-mangate.be]
> INFO 16:25:08,723 | james.mailetcontext | Storing mail
> Mail1270218307697-9822-!70969 in file://var/mail/address-error/
> INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
> ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: cleanup took 7274
> INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: cleanup took 1141
> ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: cleanup took 58100
> INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
> Slow KahaDB access: Journal append took: 10032 ms, Index update took
> 52100 ms
> INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
> Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
> 213.163.116.78 for [angelo.biddau@u-mangate.net]
> WARN 16:36:27,919 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination
> 'processor.local-address-error' - trying to recover. Cause: Java heap space
> ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> WARN 16:44:23,231 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination 'processor.spam'
> - trying to recover. Cause: Java heap space
> ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
> exception trying to create a JMSException for
> java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
> valid Object but class java.lang.String
> WARN 17:09:11,055 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination
> 'processor.local-address-error' - trying to recover. Cause: Java heap space
> ERROR 17:09:35,017 |
> org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
> Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
> could not poll endpoint:
> pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
> caused by: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> WARN 17:09:55,361 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination 'processor.spam'
> - trying to recover. Cause: Java heap space
> WARN 17:09:57,557 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination 'processor.root'
> - trying to recover. Cause: Java heap space
> WARN 17:11:00,695 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination
> 'processor.local-address-error' - trying to recover. Cause: Java heap space
> WARN 17:22:19,706 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination
> 'processor.bounces' - trying to recover. Cause: Java heap space
> WARN 17:22:42,422 |
> org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
> of JMS message listener invoker failed for destination
> 'processor.transport' - trying to recover. Cause: Java heap space
>
>
> On 04/02/2010 09:00 AM, Norman Maurer wrote:
>> Hi Eric,
>>
>> this sounds promising.... Looking forward for your next report.
>>
>> Bye
>> Norman
>>
>> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>>
>>> James survived the night.
>>> 300MB (exactly the same as when it started).
>>> No m64 file in tmp.
>>>
>>> I will post another update this evening.
>>>
>>> Tks,
>>>
>>> Eric
>>>
>>>
>>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>>
>>>> Yeah thats right.. just checout server/trunk.
>>>>
>>>> I'm looking forward to hear your feedback ;)
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>>
>>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>> Hello Norman,
>>>>> Done :)
>>>>> Tks, Eric
>>>>>
>>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>>> using.
>>>>> PS2 : I only checkout
>>>>> http://svn.apache.org/repos/asf/james/server/trunk
>>>>> and
>>>>> not the other projects. From what I read from the poms, snapshot
>>>>> version
>>>>> of
>>>>> imap-mailbox, jsieve,... are taken from the central maven repository
>>>>> and
>>>>> I
>>>>> don't have to rebuild those projects.
>>>>>
>>>>>
>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> its me again ;) Could you please svn up to latest trunk and try
>>>>>> again
>>>>>> ?
>>>>>>
>>>>>> Thx,
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>>
>>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>>
>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> I did a stress test in the last few hours (running in the
>>>>>>> background
>>>>>>> here) and I was not able to get a OOM yet with
>>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>>>>> with 5 mb.
>>>>>>>
>>>>>>> I'm using -Xmx512m and memory usage is not getting higher then
>>>>>>> 700m.
>>>>>>>
>>>>>>> Bye.
>>>>>>> Norman
>>>>>>>
>>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
nop. OutOfMemoryError is back.
I will double check my server, jdk and environment config, but other
java processes are running on the same machine without problem.
I will also have a stress scenario on my local dev PC that runs fine and
after, I will rerun it on my production server.
Dump analysis and full profiling are also to do.
Some stuff are difficult to simulate : See for example (occurs regulary)
ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
There are no more .m64 file in /tmp
Tks,
Eric
ERROR 16:24:53,636 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
INFO 16:24:58,061 | james.smtpserver | Successfully spooled mail
Mail1270218298023-9820 from jewishnessesyh2@ecs-solar.com on
68.62.205.100 for [ric.charles@u-mangate.org, ric.charles@u-mangate.com]
INFO 16:25:00,097 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: Journal append took: 989 ms, Index update took 0 ms
INFO 16:25:00,099 | james.mailetcontext | Storing mail
Mail1270218298023-9820-!226071 in file://var/mail/address-error/
INFO 16:25:07,699 | james.smtpserver | Successfully spooled mail
Mail1270218307697-9822 from oursu31@stahelin.com on 68.62.205.100 for
[ric.charles@u-mangate.be]
INFO 16:25:08,723 | james.mailetcontext | Storing mail
Mail1270218307697-9822-!70969 in file://var/mail/address-error/
INFO 16:25:09,908 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: Journal append took: 1184 ms, Index update took 0 ms
ERROR 16:25:35,144 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
INFO 16:25:41,212 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: cleanup took 7274
INFO 16:25:55,612 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: cleanup took 1141
ERROR 16:26:42,059 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: cleanup took 58100
INFO 16:27:00,003 | org.apache.activemq.store.kahadb.MessageDatabase |
Slow KahaDB access: Journal append took: 10032 ms, Index update took
52100 ms
INFO 16:27:02,183 | james.smtpserver | Successfully spooled mail
Mail1270218355606-9824 from inexperienceMO@toutgratuit.com on
213.163.116.78 for [angelo.biddau@u-mangate.net]
WARN 16:36:27,919 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination
'processor.local-address-error' - trying to recover. Cause: Java heap space
ERROR 16:43:58,281 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
WARN 16:44:23,231 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination 'processor.spam'
- trying to recover. Cause: Java heap space
ERROR 16:47:23,508 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
ERROR 16:48:37,497 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
ERROR 16:49:53,684 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
ERROR 16:52:17,970 | org.apache.activemq.ActiveMQConnection | Caught an
exception trying to create a JMSException for
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
ERROR 16:58:05,554 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
ERROR 17:04:31,427 | james.pop3server | WriteRequest holds not a an
valid Object but class java.lang.String
WARN 17:09:11,055 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination
'processor.local-address-error' - trying to recover. Cause: Java heap space
ERROR 17:09:35,017 |
org.apache.james.transport.camel.JMSSelectorPollingConsumer | Consumer
Consumer[pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000]
could not poll endpoint:
pollingjms://queue?consumer.endpointUri=activemq%3Aqueue%3Aoutgoing.retry&delay=30000
caused by: Java heap space
java.lang.OutOfMemoryError: Java heap space
WARN 17:09:55,361 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination 'processor.spam'
- trying to recover. Cause: Java heap space
WARN 17:09:57,557 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination 'processor.root'
- trying to recover. Cause: Java heap space
WARN 17:11:00,695 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination
'processor.local-address-error' - trying to recover. Cause: Java heap space
WARN 17:22:19,706 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination
'processor.bounces' - trying to recover. Cause: Java heap space
WARN 17:22:42,422 |
org.springframework.jms.listener.DefaultMessageListenerContainer | Setup
of JMS message listener invoker failed for destination
'processor.transport' - trying to recover. Cause: Java heap space
On 04/02/2010 09:00 AM, Norman Maurer wrote:
> Hi Eric,
>
> this sounds promising.... Looking forward for your next report.
>
> Bye
> Norman
>
> 2010/4/2, Eric Charles<er...@u-mangate.com>:
>
>> James survived the night.
>> 300MB (exactly the same as when it started).
>> No m64 file in tmp.
>>
>> I will post another update this evening.
>>
>> Tks,
>>
>> Eric
>>
>>
>> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>>
>>> Yeah thats right.. just checout server/trunk.
>>>
>>> I'm looking forward to hear your feedback ;)
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hello Norman,
>>>> Done :)
>>>> Tks, Eric
>>>>
>>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>>> using.
>>>> PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk
>>>> and
>>>> not the other projects. From what I read from the poms, snapshot version
>>>> of
>>>> imap-mailbox, jsieve,... are taken from the central maven repository and
>>>> I
>>>> don't have to rebuild those projects.
>>>>
>>>>
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> its me again ;) Could you please svn up to latest trunk and try again
>>>>> ?
>>>>>
>>>>> Thx,
>>>>> Norman
>>>>>
>>>>>
>>>>>
>>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>>
>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> I did a stress test in the last few hours (running in the background
>>>>>> here) and I was not able to get a OOM yet with
>>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>>>> with 5 mb.
>>>>>>
>>>>>> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>>>>>>
>>>>>> Bye.
>>>>>> Norman
>>>>>>
>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@apache.org>.
Hi Eric,
this sounds promising.... Looking forward for your next report.
Bye
Norman
2010/4/2, Eric Charles <er...@u-mangate.com>:
> James survived the night.
> 300MB (exactly the same as when it started).
> No m64 file in tmp.
>
> I will post another update this evening.
>
> Tks,
>
> Eric
>
>
> On 04/01/2010 07:06 PM, Norman Maurer wrote:
>> Yeah thats right.. just checout server/trunk.
>>
>> I'm looking forward to hear your feedback ;)
>>
>> Bye,
>> Norman
>>
>>
>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>
>>> Hello Norman,
>>> Done :)
>>> Tks, Eric
>>>
>>> PS1 : I didn't change the config, except for the JDCBDomainList I am
>>> using.
>>> PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk
>>> and
>>> not the other projects. From what I read from the poms, snapshot version
>>> of
>>> imap-mailbox, jsieve,... are taken from the central maven repository and
>>> I
>>> don't have to rebuild those projects.
>>>
>>>
>>>> Hi Eric,
>>>>
>>>> its me again ;) Could you please svn up to latest trunk and try again
>>>> ?
>>>>
>>>> Thx,
>>>> Norman
>>>>
>>>>
>>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>
>>>>> Hi Eric,
>>>>>
>>>>> I did a stress test in the last few hours (running in the background
>>>>> here) and I was not able to get a OOM yet with
>>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>>> with 5 mb.
>>>>>
>>>>> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>>>>>
>>>>> Bye.
>>>>> Norman
>>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
its me again ;) Could you please svn up to latest trunk and try again ?
Thx,
Norman
2010/4/1 Norman Maurer <no...@googlemail.com>:
> Hi Eric,
>
> I did a stress test in the last few hours (running in the background
> here) and I was not able to get a OOM yet with
> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
> with 5 mb.
>
> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>
> Bye.
> Norman
>
>
> 2010/4/1 Eric Charles <er...@u-mangate.com>:
>> Tks Stefano for the precisions.
>> I keep these in mind and already took last week some head dumps via
>> -XX:+HeapDumpOnOutOfMemoryError and jmap.
>> I have to further analyze the dumps and run full profiling this weekend.
>> For sure increasing the memory will still produce a OOM, but I have a bit
>> more time before process crash.
>>
>> Tks a lot,
>>
>> Eric
>>
>>
>> On 04/01/2010 09:44 AM, Stefano Bagnara wrote:
>>>
>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>
>>>>
>>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>>> contentOctets,
>>>> [...]
>>>> So still a OOM exception that was shown by yet-another-component (in this
>>>> case, the StoreMailbox).
>>>>
>>>
>>> OOM are shown by whichever component needs memory once the memory is
>>> exausted. So there's almost no point in taking into consideration the
>>> exception stacktrace when an OOM happens in a complex system.
>>>
>>> OOM are the results of (a) real insufficient memory (too big memory
>>> requirements), (b) memory leaks.
>>> So, either some component is configured to use more memory than the
>>> available or some component does not free resources. I guess we are in
>>> (b).
>>>
>>> So, either you go for a full profiler, or you at least take heap dumps.
>>>
>>> We have to know if memory usage grows constantly to OOM, or if you
>>> have very frequent GC that free space but then once in a while it is
>>> not enough and it throws the OOM, if the memory is full of unused
>>> objects from the same class or instead a full tree of different
>>> objects.
>>>
>>> If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
>>> jmap, jconsole are your friends here.
>>>
>>> Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
>>> so that you have an automatic heap dump on OOM (you can also set this
>>> "live" with jinfo)
>>>
>>> Also some other "guessed" information can help: the memory usage is
>>> proportional to the processed message? To their sizes? To the uptime?
>>> To the failed message..etc.
>>>
>>>
>>>>
>>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>>> job).
>>>> All 4 files were 0 bytes.
>>>>
>>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>>> max
>>>> memory).
>>>>
>>>> With the previous parameters ( -Xmx512m), the process was taking the
>>>> whole
>>>> 512MB.
>>>>
>>>
>>> Increasing the memory is rarely of help in this case: this will only
>>> help if we are in the "(a)" scenario (some component configured to use
>>> more memory than we thought). You'll probably get the OOM anyway, but
>>> it will take more time. If this happen we cat then exclude (a) and go
>>> for (b) analysis.
>>>
>>> Stefano
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
After reading the page about prefetch limits again, you should maybe
set the prefetchlimit to 1 too..
http://activemq.apache.org/what-is-the-prefetch-limit-for.html
Pooled Connections and prefetch
Consuming messages from a connection pool can be problematic due to
prefetch. Unconsumed prefetched messages are only released when a
connection is closed, but with a pooled connection the connection
close is deferred (for reuse) till the connection pool closes. This
leaves prefetched messages unconsumed till the connection is reused.
This feature can present as missing or out-of-sequence messages when
there is more than one connection in the pool.
One solution is to use pooled connections for producers and a
non-pooled connection for consumers. This might have performance
impacts on the consumer side, if multiple threads try to consume
messages at a fast rate. Alternatively, reduce the pool size to 1 for
consumers. A third alternative is to reduce the prefetchSize to 1 or 0
with the pooled connection factory. When using Spring JMS and
MessageDrivenPojo, you cannot use a prefetch of 0, so use 1 instead.
As we use a connection pool...
Bye,
Norman
2010/4/1 Norman Maurer <no...@googlemail.com>:
> Hi Eric,
>
> could you try if this helps...
>
> In spring-beans.xml replace the jmsConnectionFactory block with:
>
> <!-- jms connection pooling -->
> <bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnec
> tionFactory" destroy-method="stop">
> <property name="connectionFactory">
> <bean class="org.apache.activemq.ActiveMQConnectionFactory">
> <property name="brokerURL">
> <value>vm://localhost?broker.useJmx=false&jms.prefetchPolicy.all
> =50</value>
> </property>
> </bean>
> </property>
> </bean>
>
> Bye,
> Norman
>
> 2010/4/1 Eric Charles <er...@u-mangate.com>:
>> 2) Launching james, memory usage is 300MB. This seems normal and holding the
>> traffic well. At a certain time, memory jumps to 3GB (I gave much memory via
>> -Xmx).
>> Maybe a certain condition produces this.
>> Debugging in eclipse shows me that many threads are created by camel. I also
>> had the impression that some threads were falling and other ones were
>> recreated. If this is the case, is it normal.
>> I'm going to look at activemq/camel (I don't know them really), and also
>> still read the http://activemq.apache.org/javalangoutofmemory.html,
>> http://activemq.apache.org/my-producer-blocks.html,...
>>
>> I have the impression that everything is working fine, but at a certain
>> moment, there is a condition that makes the memory usage rise extensively
>> (not a linear growth, but really a stair).
>>
>> Bye,
>> Eric
>>
>>
>> On 04/01/2010 09:24 AM, Norman Maurer wrote:
>>>
>>> Hi Eric,
>>>
>>> thx again for your tests...
>>>
>>> So I think we have two independent problems here:
>>>
>>> 1) The .m64 files which are not deleted always:
>>>
>>> My guess here is that there is something wrong with processing Mail
>>> objects with state Null. I will investigate futher.. I also found a
>>> possible problem with .m64 files which get not deleted when a
>>> Exception is throws while processing the camel routes. I will patch
>>> this soon.
>>>
>>> 2) The OOM
>>>
>>> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>>>
>>> I think 2) is the more criticial problem and should get higher
>>> priority to get fixed..
>>>
>>> Bye,
>>> Norman
>>>
>>>
>>>
>>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>>
>>>>
>>>> Hi Norman,
>>>>
>>>> Tks for the smpt-source pointer.
>>>> I looked at it, but finally, I think I will develop those little classes
>>>> to
>>>> have more control on it.
>>>> I will put it on github.
>>>>
>>>> I launched latest trunk james yesterday evening.
>>>> During the night, 3 hours after the launch, the following occured:
>>>>
>>>> KahDB slower and slower...
>>>>
>>>> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |
>>>> Slow
>>>> KahaDB access: cleanup took 2330
>>>> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
>>>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>>>> parsed.;
>>>> nested exception is:
>>>> org.apache.james.imap.mailbox.StorageException: failed. Transaction
>>>> commit failed.;
>>>> nested exception is:
>>>> <openjpa-1.2.1-r752877:753278 fatal store error>
>>>> org.apache.openjpa.persistence.RollbackException: The transaction has
>>>> been
>>>> rolled back. See the nested exceptions for details on the errors that
>>>> occurred.
>>>> at
>>>>
>>>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>>>
>>>> at
>>>>
>>>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>>>> ... 55 more
>>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>>> contentOctets,
>>>> mediaType, subType, textu
>>>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int)
>>>> 1260,
>>>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>>>> 876]} [code=0, state=XJ001]
>>>> at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>>>> at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>>>> at
>>>>
>>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>>>> at
>>>>
>>>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>>>
>>>> So still a OOM exception that was shown by yet-another-component (in this
>>>> case, the StoreMailbox).
>>>>
>>>>
>>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>>> job).
>>>> All 4 files were 0 bytes.
>>>>
>>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>>> max
>>>> memory).
>>>>
>>>> With the previous parameters ( -Xmx512m), the process was taking the
>>>> whole
>>>> 512MB.
>>>>
>>>> Tks,
>>>> Eric
>>>>
>>>>
>>>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>>>
>>>>>
>>>>> Hi eric,
>>>>>
>>>>> thx for all your help, maybe smtp-source ( which is included in
>>>>> postfix ) can help you with stress testing.
>>>>>
>>>>> Bye
>>>>> Norman
>>>>>
>>>>> 2010/3/31, Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Norman,
>>>>>>
>>>>>> There are not so much .m64 files, so even it it shows that a mail
>>>>>> processing didn't reach the end and may have cause a leak, I don't
>>>>>> think
>>>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>>>
>>>>>> I quickly analysed a few dumps last week : there is always a class that
>>>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>>>>> (don't remember the details).
>>>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>>>> varies.
>>>>>> Today stack was:
>>>>>> org.apache.camel.RuntimeCamelException:
>>>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>>>> responseRequired = true, messageId =
>>>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>>>>> originalTransactionId = null, producerId =
>>>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>>>>>> =
>>>>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>>>> at
>>>>>>
>>>>>>
>>>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> I also think the tmp files are not the main cause.
>>>>>>
>>>>>> I have now james running in eclipse and I made already made a little
>>>>>> trip in the code. Quite impressive since last time I looked at it.
>>>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>>>> postage, but a simple class with commons-net behind) may be easier and
>>>>>> see what happens (eventually with eclipse profiler).
>>>>>>
>>>>>> It will be for this weekend for me.
>>>>>>
>>>>>> Bye,
>>>>>>
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>>>>> are not the cause of the OOM.
>>>>>>>
>>>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>>>> current trunk version), but just found to new .m64 files..
>>>>>>>
>>>>>>> So I'm still searching for the real cause. If nothing helps I will
>>>>>>> need to use a profiler to find th leak.
>>>>>>>
>>>>>>> Bye,
>>>>>>> Norman
>>>>>>>
>>>>>>> 2010/3/31 Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> I had defined the Null mailet
>>>>>>>>
>>>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>>>> <processor> local-address-error</processor>
>>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>>> </mailet>
>>>>>>>>
>>>>>>>> but now, I use the standard config
>>>>>>>>
>>>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>>>> <processor> local-address-error</processor>
>>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>>> </mailet>
>>>>>>>>
>>>>>>>> I still have the OOM.
>>>>>>>>
>>>>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>>>>> ValidRcptHandler.
>>>>>>>> I keep you posted with the result,
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> you said all the files are related to address-errors , could you
>>>>>>>>> show
>>>>>>>>> me your address error processor config?
>>>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>>>>>> the smtpserver.xml file?
>>>>>>>>>
>>>>>>>>> Thx
>>>>>>>>> Norman
>>>>>>>>>
>>>>>>>>> 2010/3/30, Eric Charles<er...@u-mangate.com>:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>
>>>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>>>>>> file, on the second after, it is no more there, so the dispose
>>>>>>>>>>> works
>>>>>>>>>>> as it should).
>>>>>>>>>>>
>>>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>>
>>>>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so
>>>>>>>>>>>> for
>>>>>>>>>>>> <un...@known.com>).
>>>>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>>>>>> thunderbird)
>>>>>>>>>>>>
>>>>>>>>>>>> However, when I sent a mail to unknown@known.com, I don't see it
>>>>>>>>>>>> in
>>>>>>>>>>>> the /tmp
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>>>>>> could
>>>>>>>>>>>>> you check somehow if the .m64 files are files which are related
>>>>>>>>>>>>> to
>>>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Norman,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>> analyse
>>>>>>>>>>>>>> it.
>>>>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come
>>>>>>>>>>>>>> back,
>>>>>>>>>>>>>> the OOM
>>>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception
>>>>>>>>>>>>>> (if
>>>>>>>>>>>>>> any,
>>>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> all
>>>>>>>>>>>>>> circumstances.
>>>>>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>>>>>> files).
>>>>>>>>>>>>>> We should also ensure after that there are no other causes to
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> leaks.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I found the cause for the not deleted temporary files.
>>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/3/29 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>>> good pointer. I will try to debug the problem later or
>>>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers..
>>>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex),
>>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but
>>>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
James survived the night.
300MB (exactly the same as when it started).
No m64 file in tmp.
I will post another update this evening.
Tks,
Eric
On 04/01/2010 07:06 PM, Norman Maurer wrote:
> Yeah thats right.. just checout server/trunk.
>
> I'm looking forward to hear your feedback ;)
>
> Bye,
> Norman
>
>
> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>
>> Hello Norman,
>> Done :)
>> Tks, Eric
>>
>> PS1 : I didn't change the config, except for the JDCBDomainList I am using.
>> PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk and
>> not the other projects. From what I read from the poms, snapshot version of
>> imap-mailbox, jsieve,... are taken from the central maven repository and I
>> don't have to rebuild those projects.
>>
>>
>>> Hi Eric,
>>>
>>> its me again ;) Could you please svn up to latest trunk and try again ?
>>>
>>> Thx,
>>> Norman
>>>
>>>
>> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>
>>>> Hi Eric,
>>>>
>>>> I did a stress test in the last few hours (running in the background
>>>> here) and I was not able to get a OOM yet with
>>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>>> with 5 mb.
>>>>
>>>> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>>>>
>>>> Bye.
>>>> Norman
>>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Yeah thats right.. just checout server/trunk.
I'm looking forward to hear your feedback ;)
Bye,
Norman
2010/4/1 Eric Charles <er...@u-mangate.com>:
> Hello Norman,
> Done :)
> Tks, Eric
>
> PS1 : I didn't change the config, except for the JDCBDomainList I am using.
> PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk and
> not the other projects. From what I read from the poms, snapshot version of
> imap-mailbox, jsieve,... are taken from the central maven repository and I
> don't have to rebuild those projects.
>
>> Hi Eric,
>>
>> its me again ;) Could you please svn up to latest trunk and try again ?
>>
>> Thx,
>> Norman
>>
> 2010/4/1 Norman Maurer<no...@googlemail.com>:
>>>
>>> Hi Eric,
>>>
>>> I did a stress test in the last few hours (running in the background
>>> here) and I was not able to get a OOM yet with
>>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>>> with 5 mb.
>>>
>>> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>>>
>>> Bye.
>>> Norman
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
Hello Norman,
Done :)
Tks, Eric
PS1 : I didn't change the config, except for the JDCBDomainList I am using.
PS2 : I only checkout http://svn.apache.org/repos/asf/james/server/trunk and not the other projects. From what I read from the poms, snapshot version of imap-mailbox, jsieve,... are taken from the central maven repository and I don't have to rebuild those projects.
> Hi Eric,
>
> its me again ;) Could you please svn up to latest trunk and try again ?
>
> Thx,
> Norman
>
2010/4/1 Norman Maurer<no...@googlemail.com>:
>>
>> Hi Eric,
>>
>> I did a stress test in the last few hours (running in the background
>> here) and I was not able to get a OOM yet with
>> "?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
>> jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
>> with 5 mb.
>>
>> I'm using -Xmx512m and memory usage is not getting higher then 700m.
>>
>> Bye.
>> Norman
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
could you try if this helps...
In spring-beans.xml replace the jmsConnectionFactory block with:
<!-- jms connection pooling -->
<bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnec
tionFactory" destroy-method="stop">
<property name="connectionFactory">
<bean class="org.apache.activemq.ActiveMQConnectionFactory">
<property name="brokerURL">
<value>vm://localhost?broker.useJmx=false&jms.prefetchPolicy.all
=50</value>
</property>
</bean>
</property>
</bean>
Bye,
Norman
2010/4/1 Eric Charles <er...@u-mangate.com>:
> 2) Launching james, memory usage is 300MB. This seems normal and holding the
> traffic well. At a certain time, memory jumps to 3GB (I gave much memory via
> -Xmx).
> Maybe a certain condition produces this.
> Debugging in eclipse shows me that many threads are created by camel. I also
> had the impression that some threads were falling and other ones were
> recreated. If this is the case, is it normal.
> I'm going to look at activemq/camel (I don't know them really), and also
> still read the http://activemq.apache.org/javalangoutofmemory.html,
> http://activemq.apache.org/my-producer-blocks.html,...
>
> I have the impression that everything is working fine, but at a certain
> moment, there is a condition that makes the memory usage rise extensively
> (not a linear growth, but really a stair).
>
> Bye,
> Eric
>
>
> On 04/01/2010 09:24 AM, Norman Maurer wrote:
>>
>> Hi Eric,
>>
>> thx again for your tests...
>>
>> So I think we have two independent problems here:
>>
>> 1) The .m64 files which are not deleted always:
>>
>> My guess here is that there is something wrong with processing Mail
>> objects with state Null. I will investigate futher.. I also found a
>> possible problem with .m64 files which get not deleted when a
>> Exception is throws while processing the camel routes. I will patch
>> this soon.
>>
>> 2) The OOM
>>
>> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>>
>> I think 2) is the more criticial problem and should get higher
>> priority to get fixed..
>>
>> Bye,
>> Norman
>>
>>
>>
>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> Tks for the smpt-source pointer.
>>> I looked at it, but finally, I think I will develop those little classes
>>> to
>>> have more control on it.
>>> I will put it on github.
>>>
>>> I launched latest trunk james yesterday evening.
>>> During the night, 3 hours after the launch, the following occured:
>>>
>>> KahDB slower and slower...
>>>
>>> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase |
>>> Slow
>>> KahaDB access: cleanup took 2330
>>> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
>>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>>> parsed.;
>>> nested exception is:
>>> org.apache.james.imap.mailbox.StorageException: failed. Transaction
>>> commit failed.;
>>> nested exception is:
>>> <openjpa-1.2.1-r752877:753278 fatal store error>
>>> org.apache.openjpa.persistence.RollbackException: The transaction has
>>> been
>>> rolled back. See the nested exceptions for details on the errors that
>>> occurred.
>>> at
>>>
>>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>>
>>> at
>>>
>>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>>> ... 55 more
>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>> contentOctets,
>>> mediaType, subType, textu
>>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int)
>>> 1260,
>>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>>> 876]} [code=0, state=XJ001]
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>>> at
>>>
>>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>>
>>> So still a OOM exception that was shown by yet-another-component (in this
>>> case, the StoreMailbox).
>>>
>>>
>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>> job).
>>> All 4 files were 0 bytes.
>>>
>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>> max
>>> memory).
>>>
>>> With the previous parameters ( -Xmx512m), the process was taking the
>>> whole
>>> 512MB.
>>>
>>> Tks,
>>> Eric
>>>
>>>
>>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi eric,
>>>>
>>>> thx for all your help, maybe smtp-source ( which is included in
>>>> postfix ) can help you with stress testing.
>>>>
>>>> Bye
>>>> Norman
>>>>
>>>> 2010/3/31, Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> There are not so much .m64 files, so even it it shows that a mail
>>>>> processing didn't reach the end and may have cause a leak, I don't
>>>>> think
>>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>>
>>>>> I quickly analysed a few dumps last week : there is always a class that
>>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>>>> (don't remember the details).
>>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>>> varies.
>>>>> Today stack was:
>>>>> org.apache.camel.RuntimeCamelException:
>>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>>> responseRequired = true, messageId =
>>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>>>> originalTransactionId = null, producerId =
>>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID
>>>>> =
>>>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>>> ...
>>>>>
>>>>>
>>>>>
>>>>> I also think the tmp files are not the main cause.
>>>>>
>>>>> I have now james running in eclipse and I made already made a little
>>>>> trip in the code. Quite impressive since last time I looked at it.
>>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>>> postage, but a simple class with commons-net behind) may be easier and
>>>>> see what happens (eventually with eclipse profiler).
>>>>>
>>>>> It will be for this weekend for me.
>>>>>
>>>>> Bye,
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>>>> are not the cause of the OOM.
>>>>>>
>>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>>> current trunk version), but just found to new .m64 files..
>>>>>>
>>>>>> So I'm still searching for the real cause. If nothing helps I will
>>>>>> need to use a profiler to find th leak.
>>>>>>
>>>>>> Bye,
>>>>>> Norman
>>>>>>
>>>>>> 2010/3/31 Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Hi Norman,
>>>>>>>
>>>>>>> I had defined the Null mailet
>>>>>>>
>>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>>> <processor> local-address-error</processor>
>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>> </mailet>
>>>>>>>
>>>>>>> but now, I use the standard config
>>>>>>>
>>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>>> <processor> local-address-error</processor>
>>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>>> </mailet>
>>>>>>>
>>>>>>> I still have the OOM.
>>>>>>>
>>>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>>>> ValidRcptHandler.
>>>>>>> I keep you posted with the result,
>>>>>>>
>>>>>>> Tks,
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> you said all the files are related to address-errors , could you
>>>>>>>> show
>>>>>>>> me your address error processor config?
>>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>>>>> the smtpserver.xml file?
>>>>>>>>
>>>>>>>> Thx
>>>>>>>> Norman
>>>>>>>>
>>>>>>>> 2010/3/30, Eric Charles<er...@u-mangate.com>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Norman,
>>>>>>>>>>
>>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>>>>> file, on the second after, it is no more there, so the dispose
>>>>>>>>>> works
>>>>>>>>>> as it should).
>>>>>>>>>>
>>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Norman,
>>>>>>>>>>>
>>>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so
>>>>>>>>>>> for
>>>>>>>>>>> <un...@known.com>).
>>>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>>>>> thunderbird)
>>>>>>>>>>>
>>>>>>>>>>> However, when I sent a mail to unknown@known.com, I don't see it
>>>>>>>>>>> in
>>>>>>>>>>> the /tmp
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>>>>> could
>>>>>>>>>>>> you check somehow if the .m64 files are files which are related
>>>>>>>>>>>> to
>>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Norman,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump
>>>>>>>>>>>>> and
>>>>>>>>>>>>> analyse
>>>>>>>>>>>>> it.
>>>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come
>>>>>>>>>>>>> back,
>>>>>>>>>>>>> the OOM
>>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception
>>>>>>>>>>>>> (if
>>>>>>>>>>>>> any,
>>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>>
>>>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed
>>>>>>>>>>>>> in
>>>>>>>>>>>>> all
>>>>>>>>>>>>> circumstances.
>>>>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>>>>> files).
>>>>>>>>>>>>> We should also ensure after that there are no other causes to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> leaks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>> Eric
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I found the cause for the not deleted temporary files.
>>>>>>>>>>>>>> Hopefully
>>>>>>>>>>>>>> this
>>>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/3/29 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> good pointer. I will try to debug the problem later or
>>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers..
>>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>>> you
>>>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex),
>>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but
>>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
2) Launching james, memory usage is 300MB. This seems normal and holding
the traffic well. At a certain time, memory jumps to 3GB (I gave much
memory via -Xmx).
Maybe a certain condition produces this.
Debugging in eclipse shows me that many threads are created by camel. I
also had the impression that some threads were falling and other ones
were recreated. If this is the case, is it normal.
I'm going to look at activemq/camel (I don't know them really), and also
still read the http://activemq.apache.org/javalangoutofmemory.html,
http://activemq.apache.org/my-producer-blocks.html,...
I have the impression that everything is working fine, but at a certain
moment, there is a condition that makes the memory usage rise
extensively (not a linear growth, but really a stair).
Bye,
Eric
On 04/01/2010 09:24 AM, Norman Maurer wrote:
> Hi Eric,
>
> thx again for your tests...
>
> So I think we have two independent problems here:
>
> 1) The .m64 files which are not deleted always:
>
> My guess here is that there is something wrong with processing Mail
> objects with state Null. I will investigate futher.. I also found a
> possible problem with .m64 files which get not deleted when a
> Exception is throws while processing the camel routes. I will patch
> this soon.
>
> 2) The OOM
>
> My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
>
> I think 2) is the more criticial problem and should get higher
> priority to get fixed..
>
> Bye,
> Norman
>
>
>
> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>
>> Hi Norman,
>>
>> Tks for the smpt-source pointer.
>> I looked at it, but finally, I think I will develop those little classes to
>> have more control on it.
>> I will put it on github.
>>
>> I launched latest trunk james yesterday evening.
>> During the night, 3 hours after the launch, the following occured:
>>
>> KahDB slower and slower...
>>
>> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
>> KahaDB access: cleanup took 2330
>> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
>> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
>> parsed.;
>> nested exception is:
>> org.apache.james.imap.mailbox.StorageException: failed. Transaction
>> commit failed.;
>> nested exception is:
>> <openjpa-1.2.1-r752877:753278 fatal store error>
>> org.apache.openjpa.persistence.RollbackException: The transaction has been
>> rolled back. See the nested exceptions for details on the errors that
>> occurred.
>> at
>> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>>
>> at
>> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
>> ... 55 more
>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
>> mediaType, subType, textu
>> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260,
>> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
>> 876]} [code=0, state=XJ001]
>> at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
>> at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
>> at
>> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
>> at
>> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>>
>> So still a OOM exception that was shown by yet-another-component (in this
>> case, the StoreMailbox).
>>
>>
>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>> job).
>> All 4 files were 0 bytes.
>>
>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
>> memory).
>>
>> With the previous parameters ( -Xmx512m), the process was taking the whole
>> 512MB.
>>
>> Tks,
>> Eric
>>
>>
>> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>
>>> Hi eric,
>>>
>>> thx for all your help, maybe smtp-source ( which is included in
>>> postfix ) can help you with stress testing.
>>>
>>> Bye
>>> Norman
>>>
>>> 2010/3/31, Eric Charles<er...@u-mangate.com>:
>>>
>>>
>>>> Hi Norman,
>>>>
>>>> There are not so much .m64 files, so even it it shows that a mail
>>>> processing didn't reach the end and may have cause a leak, I don't think
>>>> it would make crash my process so quick (between 10 and 24 hours).
>>>>
>>>> I quickly analysed a few dumps last week : there is always a class that
>>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>>> (don't remember the details).
>>>> Also notable, the stack trace (the place where the OOM gives problems)
>>>> varies.
>>>> Today stack was:
>>>> org.apache.camel.RuntimeCamelException:
>>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>>> responseRequired = true, messageId =
>>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>>> originalTransactionId = null, producerId =
>>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>>> persistent = true, type = null, priority = 4, groupID = null,
>>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>>> readOnlyBody = true, droppable = false}]. Caused by:
>>>> [java.lang.OutOfMemoryError - Java heap space]
>>>> at
>>>>
>>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>>> at
>>>>
>>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>>> ...
>>>>
>>>>
>>>>
>>>> I also think the tmp files are not the main cause.
>>>>
>>>> I have now james running in eclipse and I made already made a little
>>>> trip in the code. Quite impressive since last time I looked at it.
>>>> I will try to stress james with a small smtp/pop3 client (I used
>>>> postage, but a simple class with commons-net behind) may be easier and
>>>> see what happens (eventually with eclipse profiler).
>>>>
>>>> It will be for this weekend for me.
>>>>
>>>> Bye,
>>>>
>>>> Eric
>>>>
>>>>
>>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>>
>>>>
>>>>> Hi Eric,
>>>>>
>>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>>> are not the cause of the OOM.
>>>>>
>>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>>> current trunk version), but just found to new .m64 files..
>>>>>
>>>>> So I'm still searching for the real cause. If nothing helps I will
>>>>> need to use a profiler to find th leak.
>>>>>
>>>>> Bye,
>>>>> Norman
>>>>>
>>>>> 2010/3/31 Eric Charles<er...@u-mangate.com>:
>>>>>
>>>>>
>>>>>
>>>>>> Hi Norman,
>>>>>>
>>>>>> I had defined the Null mailet
>>>>>>
>>>>>> <mailet match="HostIsLocal" class="Null">
>>>>>> <processor> local-address-error</processor>
>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>> </mailet>
>>>>>>
>>>>>> but now, I use the standard config
>>>>>>
>>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>>> <processor> local-address-error</processor>
>>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>>> </mailet>
>>>>>>
>>>>>> I still have the OOM.
>>>>>>
>>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>>> ValidRcptHandler.
>>>>>> I keep you posted with the result,
>>>>>>
>>>>>> Tks,
>>>>>> Eric
>>>>>>
>>>>>>
>>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> you said all the files are related to address-errors , could you show
>>>>>>> me your address error processor config?
>>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>>>> the smtpserver.xml file?
>>>>>>>
>>>>>>> Thx
>>>>>>> Norman
>>>>>>>
>>>>>>> 2010/3/30, Eric Charles<er...@u-mangate.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Hi Norman,
>>>>>>>>>
>>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>>>> file, on the second after, it is no more there, so the dispose works
>>>>>>>>> as it should).
>>>>>>>>>
>>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi Norman,
>>>>>>>>>>
>>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so
>>>>>>>>>> for
>>>>>>>>>> <un...@known.com>).
>>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>>>> thunderbird)
>>>>>>>>>>
>>>>>>>>>> However, when I sent a mail to unknown@known.com, I don't see it in
>>>>>>>>>> the /tmp
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>
>>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>>>> could
>>>>>>>>>>> you check somehow if the .m64 files are files which are related to
>>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>>
>>>>>>>>>>> Thx,
>>>>>>>>>>> Norman
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Norman,
>>>>>>>>>>>>
>>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>>
>>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>>
>>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump
>>>>>>>>>>>> and
>>>>>>>>>>>> analyse
>>>>>>>>>>>> it.
>>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come
>>>>>>>>>>>> back,
>>>>>>>>>>>> the OOM
>>>>>>>>>>>> has already occured.
>>>>>>>>>>>>
>>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception
>>>>>>>>>>>> (if
>>>>>>>>>>>> any,
>>>>>>>>>>>> let's hope not).
>>>>>>>>>>>>
>>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed
>>>>>>>>>>>> in
>>>>>>>>>>>> all
>>>>>>>>>>>> circumstances.
>>>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>>>> files).
>>>>>>>>>>>> We should also ensure after that there are no other causes to the
>>>>>>>>>>>> leaks.
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> Eric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully
>>>>>>>>>>>>> this
>>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>>>> (The
>>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>>>> thats
>>>>>>>>>>>>>> a
>>>>>>>>>>>>>> good pointer. I will try to debug the problem later or
>>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>>> But I
>>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers..
>>>>>>>>>>>>>> Did
>>>>>>>>>>>>>> you
>>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and
>>>>>>>>>>>>>>> always
>>>>>>>>>>>>>>> found
>>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but
>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by
>>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
thx again for your tests...
So I think we have two independent problems here:
1) The .m64 files which are not deleted always:
My guess here is that there is something wrong with processing Mail
objects with state Null. I will investigate futher.. I also found a
possible problem with .m64 files which get not deleted when a
Exception is throws while processing the camel routes. I will patch
this soon.
2) The OOM
My guess here is that ActiveMQ is not freeing up memory. Not sure why yet
I think 2) is the more criticial problem and should get higher
priority to get fixed..
Bye,
Norman
2010/4/1 Eric Charles <er...@u-mangate.com>:
> Hi Norman,
>
> Tks for the smpt-source pointer.
> I looked at it, but finally, I think I will develop those little classes to
> have more control on it.
> I will put it on github.
>
> I launched latest trunk james yesterday evening.
> During the night, 3 hours after the launch, the following occured:
>
> KahDB slower and slower...
>
> INFO 23:37:59,884 | org.apache.activemq.store.kahadb.MessageDatabase | Slow
> KahaDB access: cleanup took 2330
> INFO 23:38:04,591 | james.mailetcontext | Error while storing mail.
> org.apache.james.imap.mailbox.MailboxException: failed. Mail cannot be
> parsed.;
> nested exception is:
> org.apache.james.imap.mailbox.StorageException: failed. Transaction
> commit failed.;
> nested exception is:
> <openjpa-1.2.1-r752877:753278 fatal store error>
> org.apache.openjpa.persistence.RollbackException: The transaction has been
> rolled back. See the nested exceptions for details on the errors that
> occurred.
> at
> org.apache.james.imap.store.StoreMailbox.appendMessage(StoreMailbox.java:265)
>
> at
> org.apache.openjpa.kernel.DelegatingStoreManager.flush(DelegatingStoreManager.java:130)
> ... 55 more
> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
> mediaType, subType, textu
> alLineCount) VALUES (?, ?, ?, ?, ?, ?, ?) [params=(long) 2678, (int) 1260,
> (byte[]) [B@7a00e70c, (long) 47887, (String) text, (String) html, (long)
> 876]} [code=0, state=XJ001]
> at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.wrap(LoggingConnectionDecorator.java:192)
> at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator.access$700(LoggingConnectionDecorator.java:57)
> at
> org.apache.openjpa.lib.jdbc.LoggingConnectionDecorator$LoggingConnection$LoggingPreparedStatement.executeUpdate(LoggingConnectionDecorator.java:866)
> at
> org.apache.openjpa.lib.jdbc.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:269)
>
> So still a OOM exception that was shown by yet-another-component (in this
> case, the StoreMailbox).
>
>
> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
> job).
> All 4 files were 0 bytes.
>
> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
> memory).
>
> With the previous parameters ( -Xmx512m), the process was taking the whole
> 512MB.
>
> Tks,
> Eric
>
>
> On 03/31/2010 08:26 PM, Norman Maurer wrote:
>>
>> Hi eric,
>>
>> thx for all your help, maybe smtp-source ( which is included in
>> postfix ) can help you with stress testing.
>>
>> Bye
>> Norman
>>
>> 2010/3/31, Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Hi Norman,
>>>
>>> There are not so much .m64 files, so even it it shows that a mail
>>> processing didn't reach the end and may have cause a leak, I don't think
>>> it would make crash my process so quick (between 10 and 24 hours).
>>>
>>> I quickly analysed a few dumps last week : there is always a class that
>>> took 30/40% of the memory. Sometimes a activemq class, sometimes not
>>> (don't remember the details).
>>> Also notable, the stack trace (the place where the OOM gives problems)
>>> varies.
>>> Today stack was:
>>> org.apache.camel.RuntimeCamelException:
>>> org.apache.camel.CamelExchangeException: Error processing Exchange.
>>> Exchange[JmsMessage: ActiveMQObjectMessage {commandId = 2102,
>>> responseRequired = true, messageId =
>>> ID:srv001-51032-1270011735798-2:0:24:1:214, originalDestination = null,
>>> originalTransactionId = null, producerId =
>>> ID:srv001-51032-1270011735798-2:0:24:1, destination =
>>> queue://processor.root, transactionId = null, expiration = 0, timestamp
>>> = 1270014325685, arrival = 0, brokerInTime = 1270014326893,
>>> brokerOutTime = 1270014333023, correlationId = null, replyTo = null,
>>> persistent = true, type = null, priority = 4, groupID = null,
>>> groupSequence = 0, targetConsumerId = null, compressed = false, userID =
>>> null, content = org.apache.activemq.util.ByteSequence@69b568d0,
>>> marshalledProperties = null, dataStructure = null, redeliveryCounter =
>>> 0, size = 10718, properties = null, readOnlyProperties = true,
>>> readOnlyBody = true, droppable = false}]. Caused by:
>>> [java.lang.OutOfMemoryError - Java heap space]
>>> at
>>>
>>> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1055)
>>> at
>>>
>>> org.apache.camel.spring.spi.TransactionErrorHandler$1.doInTransactionWithoutResult(TransactionErrorHandler.java:154)
>>> ...
>>>
>>>
>>>
>>> I also think the tmp files are not the main cause.
>>>
>>> I have now james running in eclipse and I made already made a little
>>> trip in the code. Quite impressive since last time I looked at it.
>>> I will try to stress james with a small smtp/pop3 client (I used
>>> postage, but a simple class with commons-net behind) may be easier and
>>> see what happens (eventually with eclipse profiler).
>>>
>>> It will be for this weekend for me.
>>>
>>> Bye,
>>>
>>> Eric
>>>
>>>
>>> On 03/31/2010 07:11 PM, Norman Maurer wrote:
>>>
>>>>
>>>> Hi Eric,
>>>>
>>>> thx for keeping us in the loop... I'm still not sure why the .m64
>>>> files are still in the tmp folder sometimes.. But I suspect the files
>>>> are not the cause of the OOM.
>>>>
>>>> I didn't get a OOM since yesterday morning (the time I deployed
>>>> current trunk version), but just found to new .m64 files..
>>>>
>>>> So I'm still searching for the real cause. If nothing helps I will
>>>> need to use a profiler to find th leak.
>>>>
>>>> Bye,
>>>> Norman
>>>>
>>>> 2010/3/31 Eric Charles<er...@u-mangate.com>:
>>>>
>>>>
>>>>>
>>>>> Hi Norman,
>>>>>
>>>>> I had defined the Null mailet
>>>>>
>>>>> <mailet match="HostIsLocal" class="Null">
>>>>> <processor> local-address-error</processor>
>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>> </mailet>
>>>>>
>>>>> but now, I use the standard config
>>>>>
>>>>> <mailet match="HostIsLocal" class="ToProcessor">
>>>>> <processor> local-address-error</processor>
>>>>> <notice>550 - Requested action not taken: no such user here</notice>
>>>>> </mailet>
>>>>>
>>>>> I still have the OOM.
>>>>>
>>>>> I will now deploy a fresh svn with your last commits and enable the
>>>>> ValidRcptHandler.
>>>>> I keep you posted with the result,
>>>>>
>>>>> Tks,
>>>>> Eric
>>>>>
>>>>>
>>>>> On 03/30/2010 06:49 AM, Norman Maurer wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi Eric,
>>>>>>
>>>>>> you said all the files are related to address-errors , could you show
>>>>>> me your address error processor config?
>>>>>> Does the Problem still exist when you enable the ValidRcptHandler in
>>>>>> the smtpserver.xml file?
>>>>>>
>>>>>> Thx
>>>>>> Norman
>>>>>>
>>>>>> 2010/3/30, Eric Charles<er...@u-mangate.com>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Oops, no, the files are still there (only unknown@known.com).
>>>>>>> Eric
>>>>>>>
>>>>>>>
>>>>>>> On 03/29/2010 10:16 PM, Eric Charles wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Hi Norman,
>>>>>>>>
>>>>>>>> I just deployed your new commit with the new camel DisposeProcess.
>>>>>>>> Good news : I don't see anymore the m64 file in tmp (well I see 1
>>>>>>>> file, on the second after, it is no more there, so the dispose works
>>>>>>>> as it should).
>>>>>>>>
>>>>>>>> I keep you posted with our eventual future OOM.
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 03/29/2010 09:18 PM, Eric Charles wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Norman,
>>>>>>>>>
>>>>>>>>> The .m64 are all to "unkown user" but to "well known domain" (so
>>>>>>>>> for
>>>>>>>>> <un...@known.com>).
>>>>>>>>> They are from various size (with and without attachment).
>>>>>>>>> They are well formed (I can open the downloaded file with
>>>>>>>>> thunderbird)
>>>>>>>>>
>>>>>>>>> However, when I sent a mail to unknown@known.com, I don't see it in
>>>>>>>>> the /tmp
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 03/29/2010 08:43 PM, Norman Maurer wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> sure.. we have to find the OOM cause. What would be interesting,
>>>>>>>>>> could
>>>>>>>>>> you check somehow if the .m64 files are files which are related to
>>>>>>>>>> successfully delivered mail ?
>>>>>>>>>>
>>>>>>>>>> Thx,
>>>>>>>>>> Norman
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Norman,
>>>>>>>>>>>
>>>>>>>>>>> Done. Now, we have to wait...
>>>>>>>>>>>
>>>>>>>>>>> I saw some created *.m64 that were removed.
>>>>>>>>>>> But there are other ones that remains in /tmp.
>>>>>>>>>>>
>>>>>>>>>>> I sometimes run a jmap (java memory map) to produce a heap dump
>>>>>>>>>>> and
>>>>>>>>>>> analyse
>>>>>>>>>>> it.
>>>>>>>>>>> At the beginning, everything seems ok, and often, when I come
>>>>>>>>>>> back,
>>>>>>>>>>> the OOM
>>>>>>>>>>> has already occured.
>>>>>>>>>>>
>>>>>>>>>>> I changed the -Xmx512m to -Xmx256m to have a quicker exception
>>>>>>>>>>> (if
>>>>>>>>>>> any,
>>>>>>>>>>> let's hope not).
>>>>>>>>>>>
>>>>>>>>>>> The *.m64 are a clue and we should ensure that they are removed
>>>>>>>>>>> in
>>>>>>>>>>> all
>>>>>>>>>>> circumstances.
>>>>>>>>>>> This may be the easy part, as we have some visible clues (the
>>>>>>>>>>> files).
>>>>>>>>>>> We should also ensure after that there are no other causes to the
>>>>>>>>>>> leaks.
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 03/29/2010 08:00 PM, Norman Maurer wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>
>>>>>>>>>>>> I found the cause for the not deleted temporary files. Hopefully
>>>>>>>>>>>> this
>>>>>>>>>>>> is the cause of the OOM. Could try to svn up and run again ?
>>>>>>>>>>>>
>>>>>>>>>>>> Thx,
>>>>>>>>>>>> Norman
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2010/3/29 Norman Maurer<no...@googlemail.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> thx for the report. I see exact the same problem today here..
>>>>>>>>>>>>> (The
>>>>>>>>>>>>> OOM). Didn't notice the files in the tmp folder, but I think
>>>>>>>>>>>>> thats
>>>>>>>>>>>>> a
>>>>>>>>>>>>> good pointer. I will try to debug the problem later or
>>>>>>>>>>>>> tomorrow.
>>>>>>>>>>>>> But I
>>>>>>>>>>>>> suspect you are right about the tmp files and jms producers..
>>>>>>>>>>>>> Did
>>>>>>>>>>>>> you
>>>>>>>>>>>>> run a kill -3 pid to see what threads are active etc ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thx,
>>>>>>>>>>>>> Norman
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ps: Patches are welcome :)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2010/3/29 Eric Charles<er...@u-mangate.com>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The last two weeks, I deployed various trunk snapshots.
>>>>>>>>>>>>>> After james running less than 1 day, I always ran into a
>>>>>>>>>>>>>> OutOfMemory
>>>>>>>>>>>>>> Exception.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I analysed the logs (for example STACK TRACE 1 in annex), and
>>>>>>>>>>>>>> always
>>>>>>>>>>>>>> found
>>>>>>>>>>>>>> Camel complaining when trying to deliver on JMS queue.
>>>>>>>>>>>>>> I tried to adapt the ActiveMQ configuration based on
>>>>>>>>>>>>>> http://activemq.apache.org/javalangoutofmemory.html but
>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>> helped.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I also analyzed various heap dump :sometimes, 65% was used by
>>>>>>>>>>>>>> org.apache.mina.transport.socket.nio.NioSocketSession,
>>>>>>>>>>>>>> sometimes
>>>>>>>>>>>>>> by a
>>>>>>>>>>>>>> activemq class,...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-dev-help@james.apache.org
>>>
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
I did a stress test in the last few hours (running in the background
here) and I was not able to get a OOM yet with
"?broker.useJmx=false&jms.prefetchPolicy.all=1" for the
jmsConnectionFactory. I sent 40000 emails with 100k to it and 3000
with 5 mb.
I'm using -Xmx512m and memory usage is not getting higher then 700m.
Bye.
Norman
2010/4/1 Eric Charles <er...@u-mangate.com>:
> Tks Stefano for the precisions.
> I keep these in mind and already took last week some head dumps via
> -XX:+HeapDumpOnOutOfMemoryError and jmap.
> I have to further analyze the dumps and run full profiling this weekend.
> For sure increasing the memory will still produce a OOM, but I have a bit
> more time before process crash.
>
> Tks a lot,
>
> Eric
>
>
> On 04/01/2010 09:44 AM, Stefano Bagnara wrote:
>>
>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>
>>>
>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content,
>>> contentOctets,
>>> [...]
>>> So still a OOM exception that was shown by yet-another-component (in this
>>> case, the StoreMailbox).
>>>
>>
>> OOM are shown by whichever component needs memory once the memory is
>> exausted. So there's almost no point in taking into consideration the
>> exception stacktrace when an OOM happens in a complex system.
>>
>> OOM are the results of (a) real insufficient memory (too big memory
>> requirements), (b) memory leaks.
>> So, either some component is configured to use more memory than the
>> available or some component does not free resources. I guess we are in
>> (b).
>>
>> So, either you go for a full profiler, or you at least take heap dumps.
>>
>> We have to know if memory usage grows constantly to OOM, or if you
>> have very frequent GC that free space but then once in a while it is
>> not enough and it throws the OOM, if the memory is full of unused
>> objects from the same class or instead a full tree of different
>> objects.
>>
>> If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
>> jmap, jconsole are your friends here.
>>
>> Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
>> so that you have an automatic heap dump on OOM (you can also set this
>> "live" with jinfo)
>>
>> Also some other "guessed" information can help: the memory usage is
>> proportional to the processed message? To their sizes? To the uptime?
>> To the failed message..etc.
>>
>>
>>>
>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>> job).
>>> All 4 files were 0 bytes.
>>>
>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB
>>> max
>>> memory).
>>>
>>> With the previous parameters ( -Xmx512m), the process was taking the
>>> whole
>>> 512MB.
>>>
>>
>> Increasing the memory is rarely of help in this case: this will only
>> help if we are in the "(a)" scenario (some component configured to use
>> more memory than we thought). You'll probably get the OOM anyway, but
>> it will take more time. If this happen we cat then exclude (a) and go
>> for (b) analysis.
>>
>> Stefano
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Steve Short <st...@cisco.com>.
Load a heap dump into this Memory Analyzer tool,
http://www.eclipse.org/mat/. It'll analyze the data and should list the
suspects for you.
Steve
On 01/04/2010 08:58, "Eric Charles" <er...@u-mangate.com> wrote:
> Tks Stefano for the precisions.
> I keep these in mind and already took last week some head dumps via
> -XX:+HeapDumpOnOutOfMemoryError and jmap.
> I have to further analyze the dumps and run full profiling this weekend.
> For sure increasing the memory will still produce a OOM, but I have a
> bit more time before process crash.
>
> Tks a lot,
>
> Eric
>
>
> On 04/01/2010 09:44 AM, Stefano Bagnara wrote:
>> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>>
>>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
>>> [...]
>>> So still a OOM exception that was shown by yet-another-component (in this
>>> case, the StoreMailbox).
>>>
>> OOM are shown by whichever component needs memory once the memory is
>> exausted. So there's almost no point in taking into consideration the
>> exception stacktrace when an OOM happens in a complex system.
>>
>> OOM are the results of (a) real insufficient memory (too big memory
>> requirements), (b) memory leaks.
>> So, either some component is configured to use more memory than the
>> available or some component does not free resources. I guess we are in
>> (b).
>>
>> So, either you go for a full profiler, or you at least take heap dumps.
>>
>> We have to know if memory usage grows constantly to OOM, or if you
>> have very frequent GC that free space but then once in a while it is
>> not enough and it throws the OOM, if the memory is full of unused
>> objects from the same class or instead a full tree of different
>> objects.
>>
>> If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
>> jmap, jconsole are your friends here.
>>
>> Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
>> so that you have an automatic heap dump on OOM (you can also set this
>> "live" with jinfo)
>>
>> Also some other "guessed" information can help: the memory usage is
>> proportional to the processed message? To their sizes? To the uptime?
>> To the failed message..etc.
>>
>>
>>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>>> job).
>>> All 4 files were 0 bytes.
>>>
>>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
>>> memory).
>>>
>>> With the previous parameters ( -Xmx512m), the process was taking the whole
>>> 512MB.
>>>
>> Increasing the memory is rarely of help in this case: this will only
>> help if we are in the "(a)" scenario (some component configured to use
>> more memory than we thought). You'll probably get the OOM anyway, but
>> it will take more time. If this happen we cat then exclude (a) and go
>> for (b) analysis.
>>
>> Stefano
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-dev-help@james.apache.org
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
--
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Eric Charles <er...@u-mangate.com>.
Tks Stefano for the precisions.
I keep these in mind and already took last week some head dumps via
-XX:+HeapDumpOnOutOfMemoryError and jmap.
I have to further analyze the dumps and run full profiling this weekend.
For sure increasing the memory will still produce a OOM, but I have a
bit more time before process crash.
Tks a lot,
Eric
On 04/01/2010 09:44 AM, Stefano Bagnara wrote:
> 2010/4/1 Eric Charles<er...@u-mangate.com>:
>
>> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
>> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
>> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
>> [...]
>> So still a OOM exception that was shown by yet-another-component (in this
>> case, the StoreMailbox).
>>
> OOM are shown by whichever component needs memory once the memory is
> exausted. So there's almost no point in taking into consideration the
> exception stacktrace when an OOM happens in a complex system.
>
> OOM are the results of (a) real insufficient memory (too big memory
> requirements), (b) memory leaks.
> So, either some component is configured to use more memory than the
> available or some component does not free resources. I guess we are in
> (b).
>
> So, either you go for a full profiler, or you at least take heap dumps.
>
> We have to know if memory usage grows constantly to OOM, or if you
> have very frequent GC that free space but then once in a while it is
> not enough and it throws the OOM, if the memory is full of unused
> objects from the same class or instead a full tree of different
> objects.
>
> If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
> jmap, jconsole are your friends here.
>
> Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
> so that you have an automatic heap dump on OOM (you can also set this
> "live" with jinfo)
>
> Also some other "guessed" information can help: the memory usage is
> proportional to the processed message? To their sizes? To the uptime?
> To the failed message..etc.
>
>
>> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
>> job).
>> All 4 files were 0 bytes.
>>
>> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
>> memory).
>>
>> With the previous parameters ( -Xmx512m), the process was taking the whole
>> 512MB.
>>
> Increasing the memory is rarely of help in this case: this will only
> help if we are in the "(a)" scenario (some component configured to use
> more memory than we thought). You'll probably get the OOM anyway, but
> it will take more time. If this happen we cat then exclude (a) and go
> for (b) analysis.
>
> Stefano
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: OutOfMemory exception
Posted by Stefano Bagnara <ap...@bago.org>.
2010/4/1 Eric Charles <er...@u-mangate.com>:
> Caused by: org.apache.openjpa.lib.jdbc.ReportingSQLException: Java
> exception: 'Java heap space: java.lang.OutOfMemoryError'. {prepstmnt
> 1363215207 INSERT INTO Message (id, bodyStartOctet, content, contentOctets,
> [...]
> So still a OOM exception that was shown by yet-another-component (in this
> case, the StoreMailbox).
OOM are shown by whichever component needs memory once the memory is
exausted. So there's almost no point in taking into consideration the
exception stacktrace when an OOM happens in a complex system.
OOM are the results of (a) real insufficient memory (too big memory
requirements), (b) memory leaks.
So, either some component is configured to use more memory than the
available or some component does not free resources. I guess we are in
(b).
So, either you go for a full profiler, or you at least take heap dumps.
We have to know if memory usage grows constantly to OOM, or if you
have very frequent GC that free space but then once in a while it is
not enough and it throws the OOM, if the memory is full of unused
objects from the same class or instead a full tree of different
objects.
If you don't go for a full profiler, jmap -histo, jmap -dump, jstat,
jmap, jconsole are your friends here.
Also, add the -XX:+HeapDumpOnOutOfMemoryError parameter to your jvm,
so that you have an automatic heap dump on OOM (you can also set this
"live" with jinfo)
Also some other "guessed" information can help: the memory usage is
proportional to the processed message? To their sizes? To the uptime?
To the failed message..etc.
> There were only 4 .m64 files in /tmp (the ValidRcptHandler is doing its
> job).
> All 4 files were 0 bytes.
>
> I have now launched with EXTRA_JVM_ARGUMENTS="-Xms512m -Xmx4g" (so 4GB max
> memory).
>
> With the previous parameters ( -Xmx512m), the process was taking the whole
> 512MB.
Increasing the memory is rarely of help in this case: this will only
help if we are in the "(a)" scenario (some component configured to use
more memory than we thought). You'll probably get the OOM anyway, but
it will take more time. If this happen we cat then exclude (a) and go
for (b) analysis.
Stefano
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org