You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-user@james.apache.org by Nicola Salvemini <ni...@gmail.com> on 2011/05/25 11:54:51 UTC
ActiveMQ and memory problems
Hello,
I'm using James 3.0-M3 with maildir. Everything works fine for one or
two days until I start having memory problems and I need to restart
James.
The problems begin with a long series of messages such as
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
| Failed to register MBean:
org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
and then
DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
The CPU usage is over 100% and my only alternative is to restart James.
I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
but the problem keeps coming back.
Does anyone have a suggestion to solve the problem?
Thanks in advance,
Nicola.
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Norman Maurer <no...@googlemail.com>.
Hi there,
can you give use some more informations like what java version etc.
Also if you receive "java.lang.OutOfMemoryError: PermGen space" you
need to higher the PermGen space which is done via:
-XX:MaxPermSize=256m
This hiher it to 265mb. The default is 64mb.
Bye,
Norman
2011/5/25 Nicola Salvemini <ni...@gmail.com>:
>
> Hello,
>
> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> two days until I start having memory problems and I need to restart
> James.
> The problems begin with a long series of messages such as
>
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>
> and then
>
> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
>
> The CPU usage is over 100% and my only alternative is to restart James.
> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> but the problem keeps coming back.
> Does anyone have a suggestion to solve the problem?
> Thanks in advance,
>
>
> Nicola.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hi Eric,
great!
Thanks to you, for your time and your support.
Bye,
Nicola.
2011/5/29 Eric Charles <er...@apache.org>:
> Hi Nicola,
>
> I've committed a patch.
> It could take one day for the snapshots to be available.
>
> I thank you very much for your help and the follow on this :)
> - Eric
>
> On 27/05/2011 17:38, Nicola Salvemini wrote:
>>
>> Hello Eric,
>>
>> I opened JIRA JAMES-1259.
>>
>> Thank you very much for everything, best regards,
>>
>> Nicola.
>>
>>
>>
>> On Fri, 2011-05-27 at 16:36 +0200, Eric Charles wrote:
>>>
>>> Hi Nicola,
>>>
>>> Yes, you're right.
>>>
>>> The problem arise in fact when the mail must be delivered to/cc local
>>> and non-local recipients, so when LocalDelivery and RemoteDelivery
>>> mailets must come into the play for the same mail.
>>>
>>> I see cpu rising to 100% and leaking files in blob transfer - I didn't
>>> wait to see oom, but I'm pretty sure it will come.
>>>
>>> I also tested with remotedelivery configured with a gateway and without
>>> a gateway (in my case, non gateway make the mail undeliverable to the
>>> remote recipient). This does not change anything (bug is there for both
>>> configuration).
>>>
>>> Could I ask you to open a JIRA
>>> (https://issues.apache.org/jira/browse/JAMES) for this and define it as
>>> blocker.
>>>
>>> Tks very much for the follow,
>>> - Eric
>>>
>>> On 27/05/2011 15:09, Nicola Salvemini wrote:
>>>>
>>>> Hello,
>>>> my mailetcontainer.xml is very standard, I don't use custom servlet or
>>>> something strange.
>>>> Could you please try sending only with James without using your ISP SMTP
>>>> server?
>>>>
>>>> Thanks,
>>>>
>>>> Nicola.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
>>>>>
>>>>> Replicated issue is the begin of the solution :)
>>>>>
>>>>> I sent a mail with my account in cc, and everything happens fine
>>>>> (delivered, no remaining files in blob-transfer).
>>>>>
>>>>> It goes via my ISP SMTP server before reaching James, but that should
>>>>> not change anything.
>>>>>
>>>>> Did you change mailetcontainer.xml or any other configuration ?
>>>>>
>>>>> Tks,
>>>>> - Eric
>>>>>
>>>>> On 27/05/2011 12:34, Nicola Salvemini wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have a good news! I'm able to replicate problem: I noticed the
>>>>>> problem
>>>>>> appears when James try to delivery email having a local address in CC.
>>>>>> Please consider following example:
>>>>>> James is configured as SMTP server for mydomain.com. If it try to
>>>>>> deliver email to foo@gmail.com having myaddress@mydomain.com in CC,
>>>>>> this
>>>>>> is the result:
>>>>>>
>>>>>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
>>>>>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
>>>>>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
>>>>>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor
>>>>>> transport
>>>>>>
>>>>>> Nothing more!
>>>>>>
>>>>>>> From this point on, James is not able to send any mail. All emails
>>>>>>
>>>>>> remain in "store/activemq/blob-transfer/" until OOM is reached.
>>>>>> To resolve it's necessary delete email in
>>>>>> "store/activemq/blob-transfer/" and restart the server.
>>>>>>
>>>>>> I think the problem is this. What do you think?
>>>>>>
>>>>>> Nicola.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
>>>>>>>
>>>>>>> OK, let's see what the next dump tells us.
>>>>>>>
>>>>>>> Once a OOM occurs, the consecutive exceptions can come from anywhere
>>>>>>> and
>>>>>>> should not be considered too much. Memory becomes short, jvm is
>>>>>>> confused...
>>>>>>>
>>>>>>> Have you still some messages stuck in queue and remaining files in
>>>>>>> "store/activemq/blob-transfer/" ?
>>>>>>>
>>>>>>> I was wondering about your mailetcontainer.xml configuration. Did you
>>>>>>> configure something special? Is it possible that for some reasones,
>>>>>>> the
>>>>>>> processor route a mails takes is "infinite" ?
>>>>>>>
>>>>>>> Tks,
>>>>>>> - Eric
>>>>>>>
>>>>>>>
>>>>>>> On 26/05/2011 15:18, Nicola Salvemini wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> here new exceptions found in the log. I installed Eclipe Memory
>>>>>>>> Analyzer
>>>>>>>> but I was not able to open the dump file because it was corrupt.
>>>>>>>> I'll
>>>>>>>> retry with next OOM.
>>>>>>>>
>>>>>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
>>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
>>>>>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
>>>>>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor
>>>>>>>> transport
>>>>>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
>>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully
>>>>>>>> from
>>>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
>>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully
>>>>>>>> from
>>>>>>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
>>>>>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor
>>>>>>>> for
>>>>>>>> state root reached
>>>>>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>>>> dequeue next message;
>>>>>>>> nested exception is:
>>>>>>>> javax.jms.JMSException: PermGen space
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown
>>>>>>>> Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>> at
>>>>>>>> org.springframework.jms.connection.CachingConnectionFactory
>>>>>>>>
>>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>>>> ... 2 more
>>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>>>> at java.lang.String.intern(Native Method)
>>>>>>>> at
>>>>>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
>>>>>>>> at
>>>>>>>> javax.management.ObjectName.construct(ObjectName.java:644)
>>>>>>>> at
>>>>>>>> javax.management.ObjectName.<init>(ObjectName.java:1403)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
>>>>>>>> at org.apache.activemq.broker.TransportConnection
>>>>>>>> $1.onCommand(TransportConnection.java:185)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>>>>>>> at org.apache.activemq.thread.PooledTaskRunner
>>>>>>>> $1.run(PooledTaskRunner.java:43)
>>>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>> ... 1 more
>>>>>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
>>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
>>>>>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
>>>>>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor
>>>>>>>> transport
>>>>>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
>>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully
>>>>>>>> from
>>>>>>>> abcd@abcd.org to yyyyy@zzzz.net
>>>>>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor
>>>>>>>> for
>>>>>>>> state root reached
>>>>>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
>>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
>>>>>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
>>>>>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor
>>>>>>>> transport
>>>>>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
>>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully
>>>>>>>> from
>>>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor
>>>>>>>> for
>>>>>>>> state root reached
>>>>>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
>>>>>>>> RemoteDelivery.run()
>>>>>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
>>>>>>>> RemoteDelivery.run()
>>>>>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
>>>>>>>> RemoteDelivery.run()
>>>>>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
>>>>>>>> RemoteDelivery.run()
>>>>>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail
>>>>>>>> while
>>>>>>>> spooling Unable to dequeue next message
>>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>>>> dequeue next message;
>>>>>>>> nested exception is:
>>>>>>>> javax.jms.JMSException: PermGen space
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown
>>>>>>>> Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>> at
>>>>>>>> org.springframework.jms.connection.CachingConnectionFactory
>>>>>>>>
>>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>>>> at
>>>>>>>>
>>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>>>> ... 2 more
>>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>>>>>>>>> If you go via the menu and open the dump file, it will propose you
>>>>>>>>> (after a few minutes anlaysis) a report that will present you
>>>>>>>>> "suspects". You can copy/paste this report on ml.
>>>>>>>>>
>>>>>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it
>>>>>>>>> (not
>>>>>>>>> sure it allows to analyze taken dumps, it more for live profiling).
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> - Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Eric,
>>>>>>>>>>
>>>>>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding
>>>>>>>>>> a easy
>>>>>>>>>> way to analize dump file. Have you any suggestion?
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Nicola.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> Getting OOM can have various side effects, so I bet the
>>>>>>>>>>> java.lang.VerifyError is a result of the OOM.
>>>>>>>>>>>
>>>>>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>>>>>>>>
>>>>>>>>>>> If you have now time to analyze to find the "suspect", we can do
>>>>>>>>>>> it for
>>>>>>>>>>> you if you transfer us the dump (be aware that some objects can
>>>>>>>>>>> contain
>>>>>>>>>>> some user credentials, but that really doesn't interest
>>>>>>>>>>> us).
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> - Eric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> the problem persists, but now I have a new exception in the log:
>>>>>>>>>>>>
>>>>>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing
>>>>>>>>>>>> mail while
>>>>>>>>>>>> spooling PermGen space
>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing
>>>>>>>>>>>> mail while
>>>>>>>>>>>> spooling (class: javax/mail/MessagingException, method:
>>>>>>>>>>>> superToString
>>>>>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException,
>>>>>>>>>>>> method:
>>>>>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant
>>>>>>>>>>>> pool
>>>>>>>>>>>> index
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>>>>>>>>> at
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing
>>>>>>>>>>>> mail while
>>>>>>>>>>>> spooling PermGen space
>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit
>>>>>>>>>>>>> [1] of 20/5.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If you still encounter the OOM, you can further analyse with
>>>>>>>>>>>>> "eclipse
>>>>>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you
>>>>>>>>>>>>> the
>>>>>>>>>>>>> objects that eat memory (launch james with standard Mx 512M,
>>>>>>>>>>>>> otherwise
>>>>>>>>>>>>> you will need more time to laod dump).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>>>>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] http://s.apache.org/30x
>>>>>>>>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>>> I'm using snapshot
>>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> now I'll try to install snapshot
>>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>>>>>>>>> and I'll add the complete debugging path in the startup script
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>> further analyse the dump.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Many thanks for your help,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tks for your answer.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>>>>>>>>> - maildir
>>>>>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but
>>>>>>>>>>>>>>> it's rejected
>>>>>>>>>>>>>>> via smtp hook)
>>>>>>>>>>>>>>> - not many users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My config here is really stable after weeks (no oom, low
>>>>>>>>>>>>>>> memory
>>>>>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders
>>>>>>>>>>>>>>> empty).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the
>>>>>>>>>>>>>>> issue.
>>>>>>>>>>>>>>> Would it come from recent
>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>>>>>>>>> btw, which version are you using ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The complete debugging path would be to add
>>>>>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path
>>>>>>>>>>>>>>> in the
>>>>>>>>>>>>>>> startup script and to further analyse the dump: it will give
>>>>>>>>>>>>>>> for sure
>>>>>>>>>>>>>>> the cause of oom.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> at this time we don't have very high mail traffic, more or
>>>>>>>>>>>>>>>> less 500/600
>>>>>>>>>>>>>>>> mail per day.
>>>>>>>>>>>>>>>> James is configured as smtp and imap servers and I think
>>>>>>>>>>>>>>>> that most of
>>>>>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active
>>>>>>>>>>>>>>>> clients at the
>>>>>>>>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that
>>>>>>>>>>>>>>>>> issue?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846
>>>>>>>>>>>>>>>> Got<tag>: 1125
>>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846
>>>>>>>>>>>>>>>> Got<command>:
>>>>>>>>>>>>>>>> noop
>>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601
>>>>>>>>>>>>>>>> executing hook
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352
>>>>>>>>>>>>>>>> Got<tag>: A05974
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage"
>>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled
>>>>>>>>>>>>>>>> mail will be
>>>>>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/"
>>>>>>>>>>>>>>>> folder. If
>>>>>>>>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Last thing: James is installed on a server machine whit
>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>> characteristics:
>>>>>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel
>>>>>>>>>>>>>>>> 2.6.37.6-0.5
>>>>>>>>>>>>>>>> java version: 1.6.0_25
>>>>>>>>>>>>>>>> RAM: 8Gb
>>>>>>>>>>>>>>>> Architecture: x86_64
>>>>>>>>>>>>>>>> CPU(s): 6
>>>>>>>>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>>>>>>>>> Thread(s) per core: 1
>>>>>>>>>>>>>>>> Core(s) per socket: 6
>>>>>>>>>>>>>>>> CPU socket(s): 1
>>>>>>>>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>>>>>>>>> CPU family: 16
>>>>>>>>>>>>>>>> Model: 8
>>>>>>>>>>>>>>>> Stepping: 1
>>>>>>>>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for your help,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine
>>>>>>>>>>>>>>>>>> for one or
>>>>>>>>>>>>>>>>>> two days until I start having memory problems and I need
>>>>>>>>>>>>>>>>>> to restart
>>>>>>>>>>>>>>>>>> James.
>>>>>>>>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to
>>>>>>>>>>>>>>>>>> restart James.
>>>>>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have
>>>>>>>>>>>>>>>>>> -Xms512m -Xmx1024m)
>>>>>>>>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Hi Nicola,
I've committed a patch.
It could take one day for the snapshots to be available.
I thank you very much for your help and the follow on this :)
- Eric
On 27/05/2011 17:38, Nicola Salvemini wrote:
> Hello Eric,
>
> I opened JIRA JAMES-1259.
>
> Thank you very much for everything, best regards,
>
> Nicola.
>
>
>
> On Fri, 2011-05-27 at 16:36 +0200, Eric Charles wrote:
>> Hi Nicola,
>>
>> Yes, you're right.
>>
>> The problem arise in fact when the mail must be delivered to/cc local
>> and non-local recipients, so when LocalDelivery and RemoteDelivery
>> mailets must come into the play for the same mail.
>>
>> I see cpu rising to 100% and leaking files in blob transfer - I didn't
>> wait to see oom, but I'm pretty sure it will come.
>>
>> I also tested with remotedelivery configured with a gateway and without
>> a gateway (in my case, non gateway make the mail undeliverable to the
>> remote recipient). This does not change anything (bug is there for both
>> configuration).
>>
>> Could I ask you to open a JIRA
>> (https://issues.apache.org/jira/browse/JAMES) for this and define it as
>> blocker.
>>
>> Tks very much for the follow,
>> - Eric
>>
>> On 27/05/2011 15:09, Nicola Salvemini wrote:
>>> Hello,
>>> my mailetcontainer.xml is very standard, I don't use custom servlet or
>>> something strange.
>>> Could you please try sending only with James without using your ISP SMTP
>>> server?
>>>
>>> Thanks,
>>>
>>> Nicola.
>>>
>>>
>>>
>>>
>>> On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
>>>> Replicated issue is the begin of the solution :)
>>>>
>>>> I sent a mail with my account in cc, and everything happens fine
>>>> (delivered, no remaining files in blob-transfer).
>>>>
>>>> It goes via my ISP SMTP server before reaching James, but that should
>>>> not change anything.
>>>>
>>>> Did you change mailetcontainer.xml or any other configuration ?
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>> On 27/05/2011 12:34, Nicola Salvemini wrote:
>>>>> Hello,
>>>>>
>>>>> I have a good news! I'm able to replicate problem: I noticed the problem
>>>>> appears when James try to delivery email having a local address in CC.
>>>>> Please consider following example:
>>>>> James is configured as SMTP server for mydomain.com. If it try to
>>>>> deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
>>>>> is the result:
>>>>>
>>>>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
>>>>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
>>>>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
>>>>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
>>>>>
>>>>> Nothing more!
>>>>>
>>>>>> From this point on, James is not able to send any mail. All emails
>>>>> remain in "store/activemq/blob-transfer/" until OOM is reached.
>>>>> To resolve it's necessary delete email in
>>>>> "store/activemq/blob-transfer/" and restart the server.
>>>>>
>>>>> I think the problem is this. What do you think?
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
>>>>>> OK, let's see what the next dump tells us.
>>>>>>
>>>>>> Once a OOM occurs, the consecutive exceptions can come from anywhere and
>>>>>> should not be considered too much. Memory becomes short, jvm is confused...
>>>>>>
>>>>>> Have you still some messages stuck in queue and remaining files in
>>>>>> "store/activemq/blob-transfer/" ?
>>>>>>
>>>>>> I was wondering about your mailetcontainer.xml configuration. Did you
>>>>>> configure something special? Is it possible that for some reasones, the
>>>>>> processor route a mails takes is "infinite" ?
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>>
>>>>>> On 26/05/2011 15:18, Nicola Salvemini wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> here new exceptions found in the log. I installed Eclipe Memory Analyzer
>>>>>>> but I was not able to open the dump file because it was corrupt. I'll
>>>>>>> retry with next OOM.
>>>>>>>
>>>>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
>>>>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
>>>>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
>>>>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
>>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>>>>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
>>>>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
>>>>>>> state root reached
>>>>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>>> dequeue next message;
>>>>>>> nested exception is:
>>>>>>> javax.jms.JMSException: PermGen space
>>>>>>> at
>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>>> at
>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>>> at
>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.springframework.jms.connection.CachingConnectionFactory
>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>>> at
>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>>> ... 2 more
>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>>> at java.lang.String.intern(Native Method)
>>>>>>> at
>>>>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
>>>>>>> at javax.management.ObjectName.construct(ObjectName.java:644)
>>>>>>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>> at
>>>>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
>>>>>>> at
>>>>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
>>>>>>> at
>>>>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
>>>>>>> at org.apache.activemq.broker.TransportConnection
>>>>>>> $1.onCommand(TransportConnection.java:185)
>>>>>>> at
>>>>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>>>>> at
>>>>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>>>>>> at
>>>>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
>>>>>>> at
>>>>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>>>>>> at org.apache.activemq.thread.PooledTaskRunner
>>>>>>> $1.run(PooledTaskRunner.java:43)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>>>> ... 1 more
>>>>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
>>>>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
>>>>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
>>>>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
>>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
>>>>>>> abcd@abcd.org to yyyyy@zzzz.net
>>>>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
>>>>>>> state root reached
>>>>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
>>>>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
>>>>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
>>>>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
>>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
>>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
>>>>>>> state root reached
>>>>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
>>>>>>> RemoteDelivery.run()
>>>>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
>>>>>>> RemoteDelivery.run()
>>>>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
>>>>>>> RemoteDelivery.run()
>>>>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
>>>>>>> RemoteDelivery.run()
>>>>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
>>>>>>> spooling Unable to dequeue next message
>>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>>> dequeue next message;
>>>>>>> nested exception is:
>>>>>>> javax.jms.JMSException: PermGen space
>>>>>>> at
>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>>> at
>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>>> at
>>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>>> at
>>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.springframework.jms.connection.CachingConnectionFactory
>>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>>> at
>>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>>> ... 2 more
>>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>>>>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>>>>>>>> If you go via the menu and open the dump file, it will propose you
>>>>>>>> (after a few minutes anlaysis) a report that will present you
>>>>>>>> "suspects". You can copy/paste this report on ml.
>>>>>>>>
>>>>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
>>>>>>>> sure it allows to analyze taken dumps, it more for live profiling).
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> - Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>>>>>>>> Hi Eric,
>>>>>>>>>
>>>>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
>>>>>>>>> way to analize dump file. Have you any suggestion?
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Nicola.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>>>>>>>> Hi,
>>>>>>>>>> Getting OOM can have various side effects, so I bet the
>>>>>>>>>> java.lang.VerifyError is a result of the OOM.
>>>>>>>>>>
>>>>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>>>>>>>
>>>>>>>>>> If you have now time to analyze to find the "suspect", we can do it for
>>>>>>>>>> you if you transfer us the dump (be aware that some objects can contain
>>>>>>>>>> some user credentials, but that really doesn't interest us).
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> - Eric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> the problem persists, but now I have a new exception in the log:
>>>>>>>>>>>
>>>>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
>>>>>>>>>>> spooling PermGen space
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
>>>>>>>>>>> spooling (class: javax/mail/MessagingException, method: superToString
>>>>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
>>>>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
>>>>>>>>>>> index
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
>>>>>>>>>>> spooling PermGen space
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nicola.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>>>>>>>>>>>
>>>>>>>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
>>>>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>>>>>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
>>>>>>>>>>>> you will need more time to laod dump).
>>>>>>>>>>>>
>>>>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>>>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> - Eric
>>>>>>>>>>>>
>>>>>>>>>>>> [1] http://s.apache.org/30x
>>>>>>>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>>>>>>>
>>>>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>>> I'm using snapshot
>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>>>>>>>
>>>>>>>>>>>>> now I'll try to install snapshot
>>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>>>>>>>> and I'll add the complete debugging path in the startup script to
>>>>>>>>>>>>> further analyse the dump.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Many thanks for your help,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>>>>>>>> Tks for your answer.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>>>>>>>> - maildir
>>>>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>>>>>>>>>>>> via smtp hook)
>>>>>>>>>>>>>> - not many users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>>>>>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>>>>>>>> btw, which version are you using ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The complete debugging path would be to add
>>>>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>>>>>>>>>>>> startup script and to further analyse the dump: it will give for sure
>>>>>>>>>>>>>> the cause of oom.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>>>>>>>>>>>> mail per day.
>>>>>>>>>>>>>>> James is configured as smtp and imap servers and I think that most of
>>>>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>>>>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>>>>>>>>>>>> noop
>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>>>>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Last thing: James is installed on a server machine whit following
>>>>>>>>>>>>>>> characteristics:
>>>>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>>>>>>>>>>>> java version: 1.6.0_25
>>>>>>>>>>>>>>> RAM: 8Gb
>>>>>>>>>>>>>>> Architecture: x86_64
>>>>>>>>>>>>>>> CPU(s): 6
>>>>>>>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>>>>>>>> Thread(s) per core: 1
>>>>>>>>>>>>>>> Core(s) per socket: 6
>>>>>>>>>>>>>>> CPU socket(s): 1
>>>>>>>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>>>>>>>> CPU family: 16
>>>>>>>>>>>>>>> Model: 8
>>>>>>>>>>>>>>> Stepping: 1
>>>>>>>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for your help,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>>>>>>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>>>>>>>>>>>> James.
>>>>>>>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hello Eric,
I opened JIRA JAMES-1259.
Thank you very much for everything, best regards,
Nicola.
On Fri, 2011-05-27 at 16:36 +0200, Eric Charles wrote:
> Hi Nicola,
>
> Yes, you're right.
>
> The problem arise in fact when the mail must be delivered to/cc local
> and non-local recipients, so when LocalDelivery and RemoteDelivery
> mailets must come into the play for the same mail.
>
> I see cpu rising to 100% and leaking files in blob transfer - I didn't
> wait to see oom, but I'm pretty sure it will come.
>
> I also tested with remotedelivery configured with a gateway and without
> a gateway (in my case, non gateway make the mail undeliverable to the
> remote recipient). This does not change anything (bug is there for both
> configuration).
>
> Could I ask you to open a JIRA
> (https://issues.apache.org/jira/browse/JAMES) for this and define it as
> blocker.
>
> Tks very much for the follow,
> - Eric
>
> On 27/05/2011 15:09, Nicola Salvemini wrote:
> > Hello,
> > my mailetcontainer.xml is very standard, I don't use custom servlet or
> > something strange.
> > Could you please try sending only with James without using your ISP SMTP
> > server?
> >
> > Thanks,
> >
> > Nicola.
> >
> >
> >
> >
> > On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
> >> Replicated issue is the begin of the solution :)
> >>
> >> I sent a mail with my account in cc, and everything happens fine
> >> (delivered, no remaining files in blob-transfer).
> >>
> >> It goes via my ISP SMTP server before reaching James, but that should
> >> not change anything.
> >>
> >> Did you change mailetcontainer.xml or any other configuration ?
> >>
> >> Tks,
> >> - Eric
> >>
> >> On 27/05/2011 12:34, Nicola Salvemini wrote:
> >>> Hello,
> >>>
> >>> I have a good news! I'm able to replicate problem: I noticed the problem
> >>> appears when James try to delivery email having a local address in CC.
> >>> Please consider following example:
> >>> James is configured as SMTP server for mydomain.com. If it try to
> >>> deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
> >>> is the result:
> >>>
> >>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
> >>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
> >>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
> >>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
> >>>
> >>> Nothing more!
> >>>
> >>>> From this point on, James is not able to send any mail. All emails
> >>> remain in "store/activemq/blob-transfer/" until OOM is reached.
> >>> To resolve it's necessary delete email in
> >>> "store/activemq/blob-transfer/" and restart the server.
> >>>
> >>> I think the problem is this. What do you think?
> >>>
> >>> Nicola.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
> >>>> OK, let's see what the next dump tells us.
> >>>>
> >>>> Once a OOM occurs, the consecutive exceptions can come from anywhere and
> >>>> should not be considered too much. Memory becomes short, jvm is confused...
> >>>>
> >>>> Have you still some messages stuck in queue and remaining files in
> >>>> "store/activemq/blob-transfer/" ?
> >>>>
> >>>> I was wondering about your mailetcontainer.xml configuration. Did you
> >>>> configure something special? Is it possible that for some reasones, the
> >>>> processor route a mails takes is "infinite" ?
> >>>>
> >>>> Tks,
> >>>> - Eric
> >>>>
> >>>>
> >>>> On 26/05/2011 15:18, Nicola Salvemini wrote:
> >>>>> Hi,
> >>>>>
> >>>>> here new exceptions found in the log. I installed Eclipe Memory Analyzer
> >>>>> but I was not able to open the dump file because it was corrupt. I'll
> >>>>> retry with next OOM.
> >>>>>
> >>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
> >>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
> >>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
> >>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
> >>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
> >>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> >>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
> >>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
> >>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> >>>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
> >>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
> >>>>> state root reached
> >>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> >>>>> dequeue next message;
> >>>>> nested exception is:
> >>>>> javax.jms.JMSException: PermGen space
> >>>>> at
> >>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> >>>>> at
> >>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> Caused by: javax.jms.JMSException: PermGen space
> >>>>> at
> >>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> >>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> >>>>> at
> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>> at org.springframework.jms.connection.CachingConnectionFactory
> >>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> >>>>> at $Proxy61.createConsumer(Unknown Source)
> >>>>> at
> >>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> >>>>> ... 2 more
> >>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
> >>>>> at java.lang.String.intern(Native Method)
> >>>>> at
> >>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
> >>>>> at javax.management.ObjectName.construct(ObjectName.java:644)
> >>>>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
> >>>>> at
> >>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
> >>>>> at
> >>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
> >>>>> at
> >>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
> >>>>> at
> >>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
> >>>>> at
> >>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
> >>>>> at
> >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>>>> at
> >>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
> >>>>> at
> >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>>>> at
> >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>>>> at
> >>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>>>> at
> >>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
> >>>>> at
> >>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
> >>>>> at
> >>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
> >>>>> at
> >>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> >>>>> at org.apache.activemq.broker.TransportConnection
> >>>>> $1.onCommand(TransportConnection.java:185)
> >>>>> at
> >>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >>>>> at
> >>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> >>>>> at
> >>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
> >>>>> at
> >>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> >>>>> at org.apache.activemq.thread.PooledTaskRunner
> >>>>> $1.run(PooledTaskRunner.java:43)
> >>>>> at java.util.concurrent.ThreadPoolExecutor
> >>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>> at java.util.concurrent.ThreadPoolExecutor
> >>>>> $Worker.run(ThreadPoolExecutor.java:908)
> >>>>> ... 1 more
> >>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
> >>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
> >>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
> >>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
> >>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
> >>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
> >>>>> abcd@abcd.org to yyyyy@zzzz.net
> >>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
> >>>>> state root reached
> >>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
> >>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
> >>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
> >>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
> >>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
> >>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
> >>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
> >>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
> >>>>> state root reached
> >>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
> >>>>> RemoteDelivery.run()
> >>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
> >>>>> RemoteDelivery.run()
> >>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
> >>>>> RemoteDelivery.run()
> >>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
> >>>>> RemoteDelivery.run()
> >>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
> >>>>> spooling Unable to dequeue next message
> >>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> >>>>> dequeue next message;
> >>>>> nested exception is:
> >>>>> javax.jms.JMSException: PermGen space
> >>>>> at
> >>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> >>>>> at
> >>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> Caused by: javax.jms.JMSException: PermGen space
> >>>>> at
> >>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> >>>>> at
> >>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> >>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> >>>>> at
> >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>>>> at org.springframework.jms.connection.CachingConnectionFactory
> >>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> >>>>> at $Proxy61.createConsumer(Unknown Source)
> >>>>> at
> >>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> >>>>> ... 2 more
> >>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
> >>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
> >>>>>> If you go via the menu and open the dump file, it will propose you
> >>>>>> (after a few minutes anlaysis) a report that will present you
> >>>>>> "suspects". You can copy/paste this report on ml.
> >>>>>>
> >>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
> >>>>>> sure it allows to analyze taken dumps, it more for live profiling).
> >>>>>>
> >>>>>> Tks,
> >>>>>> - Eric
> >>>>>>
> >>>>>>
> >>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
> >>>>>>> Hi Eric,
> >>>>>>>
> >>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
> >>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
> >>>>>>> way to analize dump file. Have you any suggestion?
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Nicola.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
> >>>>>>>> Hi,
> >>>>>>>> Getting OOM can have various side effects, so I bet the
> >>>>>>>> java.lang.VerifyError is a result of the OOM.
> >>>>>>>>
> >>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
> >>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
> >>>>>>>>
> >>>>>>>> If you have now time to analyze to find the "suspect", we can do it for
> >>>>>>>> you if you transfer us the dump (be aware that some objects can contain
> >>>>>>>> some user credentials, but that really doesn't interest us).
> >>>>>>>>
> >>>>>>>> Tks,
> >>>>>>>> - Eric
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> the problem persists, but now I have a new exception in the log:
> >>>>>>>>>
> >>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> >>>>>>>>> spooling PermGen space
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> >>>>>>>>> spooling (class: javax/mail/MessagingException, method: superToString
> >>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
> >>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> >>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
> >>>>>>>>> index
> >>>>>>>>> at
> >>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> >>>>>>>>> at
> >>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> >>>>>>>>> at
> >>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> >>>>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> >>>>>>>>> spooling PermGen space
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Nicola.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> >>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
> >>>>>>>>>>
> >>>>>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
> >>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
> >>>>>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
> >>>>>>>>>> you will need more time to laod dump).
> >>>>>>>>>>
> >>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
> >>>>>>>>>> james-server-context.xml (see changes from [1]).
> >>>>>>>>>>
> >>>>>>>>>> Tks,
> >>>>>>>>>> - Eric
> >>>>>>>>>>
> >>>>>>>>>> [1] http://s.apache.org/30x
> >>>>>>>>>> [2] http://www.eclipse.org/mat/
> >>>>>>>>>>
> >>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
> >>>>>>>>>>> Hi Eric,
> >>>>>>>>>>> I'm using snapshot
> >>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
> >>>>>>>>>>>
> >>>>>>>>>>> now I'll try to install snapshot
> >>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
> >>>>>>>>>>> and I'll add the complete debugging path in the startup script to
> >>>>>>>>>>> further analyse the dump.
> >>>>>>>>>>>
> >>>>>>>>>>> Many thanks for your help,
> >>>>>>>>>>>
> >>>>>>>>>>> Nicola.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >>>>>>>>>>>> Tks for your answer.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So you've got more or less the same configuration as I have:
> >>>>>>>>>>>> - maildir
> >>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >>>>>>>>>>>> via smtp hook)
> >>>>>>>>>>>> - not many users
> >>>>>>>>>>>>
> >>>>>>>>>>>> My config here is really stable after weeks (no oom, low memory
> >>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
> >>>>>>>>>>>>
> >>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
> >>>>>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >>>>>>>>>>>> btw, which version are you using ?
> >>>>>>>>>>>>
> >>>>>>>>>>>> The complete debugging path would be to add
> >>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >>>>>>>>>>>> startup script and to further analyse the dump: it will give for sure
> >>>>>>>>>>>> the cause of oom.
> >>>>>>>>>>>>
> >>>>>>>>>>>> - Eric
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
> >>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
> >>>>>>>>>>>>> mail per day.
> >>>>>>>>>>>>> James is configured as smtp and imap servers and I think that most of
> >>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
> >>>>>>>>>>>>> same time making requests to the James IMAP server).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> This will allow us to better define the place to look at.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, it is in most cases.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is the log still showing messages after the OOM?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Yes, please consider following log sequence:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>>>>>>>>>>>> noop
> >>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
> >>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>>>>>>>>>>>> the number of mail rises too high then the problem occurs.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Last thing: James is installed on a server machine whit following
> >>>>>>>>>>>>> characteristics:
> >>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>>>>>>>>>>>> java version: 1.6.0_25
> >>>>>>>>>>>>> RAM: 8Gb
> >>>>>>>>>>>>> Architecture: x86_64
> >>>>>>>>>>>>> CPU(s): 6
> >>>>>>>>>>>>> On-line CPU(s) list: 0-5
> >>>>>>>>>>>>> Thread(s) per core: 1
> >>>>>>>>>>>>> Core(s) per socket: 6
> >>>>>>>>>>>>> CPU socket(s): 1
> >>>>>>>>>>>>> Vendor ID: AuthenticAMD
> >>>>>>>>>>>>> CPU family: 16
> >>>>>>>>>>>>> Model: 8
> >>>>>>>>>>>>> Stepping: 1
> >>>>>>>>>>>>> CPU MHz: 2599.966
> >>>>>>>>>>>>> BogoMIPS: 5200.12
> >>>>>>>>>>>>> Virtualization: AMD-V
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thanks for your help,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Nicola.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Tks,
> >>>>>>>>>>>>>> - Eric
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>>>>>>>>>>>> two days until I start having memory problems and I need to restart
> >>>>>>>>>>>>>>> James.
> >>>>>>>>>>>>>>> The problems begin with a long series of messages such as
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> and then
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>>>>>>>>>>>> but the problem keeps coming back.
> >>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
> >>>>>>>>>>>>>>> Thanks in advance,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Nicola.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Norman Maurer <no...@googlemail.com>.
Hi Eric,
could you try to downgrade to camel 2.5.0 and see if you can still
reproduce it ?
Thanks,
Norman
2011/5/27 Eric Charles <er...@apache.org>:
> Hi Nicola,
>
> Yes, you're right.
>
> The problem arise in fact when the mail must be delivered to/cc local and
> non-local recipients, so when LocalDelivery and RemoteDelivery mailets must
> come into the play for the same mail.
>
> I see cpu rising to 100% and leaking files in blob transfer - I didn't wait
> to see oom, but I'm pretty sure it will come.
>
> I also tested with remotedelivery configured with a gateway and without a
> gateway (in my case, non gateway make the mail undeliverable to the remote
> recipient). This does not change anything (bug is there for both
> configuration).
>
> Could I ask you to open a JIRA (https://issues.apache.org/jira/browse/JAMES)
> for this and define it as blocker.
>
> Tks very much for the follow,
> - Eric
>
> On 27/05/2011 15:09, Nicola Salvemini wrote:
>>
>> Hello,
>> my mailetcontainer.xml is very standard, I don't use custom servlet or
>> something strange.
>> Could you please try sending only with James without using your ISP SMTP
>> server?
>>
>> Thanks,
>>
>> Nicola.
>>
>>
>>
>>
>> On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
>>>
>>> Replicated issue is the begin of the solution :)
>>>
>>> I sent a mail with my account in cc, and everything happens fine
>>> (delivered, no remaining files in blob-transfer).
>>>
>>> It goes via my ISP SMTP server before reaching James, but that should
>>> not change anything.
>>>
>>> Did you change mailetcontainer.xml or any other configuration ?
>>>
>>> Tks,
>>> - Eric
>>>
>>> On 27/05/2011 12:34, Nicola Salvemini wrote:
>>>>
>>>> Hello,
>>>>
>>>> I have a good news! I'm able to replicate problem: I noticed the problem
>>>> appears when James try to delivery email having a local address in CC.
>>>> Please consider following example:
>>>> James is configured as SMTP server for mydomain.com. If it try to
>>>> deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
>>>> is the result:
>>>>
>>>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
>>>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
>>>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
>>>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
>>>>
>>>> Nothing more!
>>>>
>>>>> From this point on, James is not able to send any mail. All emails
>>>>
>>>> remain in "store/activemq/blob-transfer/" until OOM is reached.
>>>> To resolve it's necessary delete email in
>>>> "store/activemq/blob-transfer/" and restart the server.
>>>>
>>>> I think the problem is this. What do you think?
>>>>
>>>> Nicola.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
>>>>>
>>>>> OK, let's see what the next dump tells us.
>>>>>
>>>>> Once a OOM occurs, the consecutive exceptions can come from anywhere
>>>>> and
>>>>> should not be considered too much. Memory becomes short, jvm is
>>>>> confused...
>>>>>
>>>>> Have you still some messages stuck in queue and remaining files in
>>>>> "store/activemq/blob-transfer/" ?
>>>>>
>>>>> I was wondering about your mailetcontainer.xml configuration. Did you
>>>>> configure something special? Is it possible that for some reasones, the
>>>>> processor route a mails takes is "infinite" ?
>>>>>
>>>>> Tks,
>>>>> - Eric
>>>>>
>>>>>
>>>>> On 26/05/2011 15:18, Nicola Salvemini wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> here new exceptions found in the log. I installed Eclipe Memory
>>>>>> Analyzer
>>>>>> but I was not able to open the dump file because it was corrupt. I'll
>>>>>> retry with next OOM.
>>>>>>
>>>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
>>>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
>>>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor
>>>>>> transport
>>>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully
>>>>>> from
>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
>>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully
>>>>>> from
>>>>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
>>>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
>>>>>> state root reached
>>>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>> dequeue next message;
>>>>>> nested exception is:
>>>>>> javax.jms.JMSException: PermGen space
>>>>>> at
>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown
>>>>>> Source)
>>>>>> at
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.springframework.jms.connection.CachingConnectionFactory
>>>>>>
>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>> at
>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>> ... 2 more
>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>> at java.lang.String.intern(Native Method)
>>>>>> at
>>>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
>>>>>> at
>>>>>> javax.management.ObjectName.construct(ObjectName.java:644)
>>>>>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
>>>>>> at
>>>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
>>>>>> at org.apache.activemq.broker.TransportConnection
>>>>>> $1.onCommand(TransportConnection.java:185)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>>>>> at org.apache.activemq.thread.PooledTaskRunner
>>>>>> $1.run(PooledTaskRunner.java:43)
>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>>> ... 1 more
>>>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
>>>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
>>>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor
>>>>>> transport
>>>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
>>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully
>>>>>> from
>>>>>> abcd@abcd.org to yyyyy@zzzz.net
>>>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
>>>>>> state root reached
>>>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
>>>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
>>>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor
>>>>>> transport
>>>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
>>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully
>>>>>> from
>>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
>>>>>> state root reached
>>>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
>>>>>> RemoteDelivery.run()
>>>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
>>>>>> RemoteDelivery.run()
>>>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
>>>>>> RemoteDelivery.run()
>>>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
>>>>>> RemoteDelivery.run()
>>>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail
>>>>>> while
>>>>>> spooling Unable to dequeue next message
>>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>>> dequeue next message;
>>>>>> nested exception is:
>>>>>> javax.jms.JMSException: PermGen space
>>>>>> at
>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>>> at
>>>>>>
>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>>> at
>>>>>>
>>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown
>>>>>> Source)
>>>>>> at
>>>>>>
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at
>>>>>> org.springframework.jms.connection.CachingConnectionFactory
>>>>>>
>>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>>> at
>>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>>> ... 2 more
>>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>>>>>>>
>>>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>>>>>>> If you go via the menu and open the dump file, it will propose you
>>>>>>> (after a few minutes anlaysis) a report that will present you
>>>>>>> "suspects". You can copy/paste this report on ml.
>>>>>>>
>>>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it
>>>>>>> (not
>>>>>>> sure it allows to analyze taken dumps, it more for live profiling).
>>>>>>>
>>>>>>> Tks,
>>>>>>> - Eric
>>>>>>>
>>>>>>>
>>>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>>>>>>>
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a
>>>>>>>> easy
>>>>>>>> way to analize dump file. Have you any suggestion?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nicola.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Getting OOM can have various side effects, so I bet the
>>>>>>>>> java.lang.VerifyError is a result of the OOM.
>>>>>>>>>
>>>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>>>>>>
>>>>>>>>> If you have now time to analyze to find the "suspect", we can do it
>>>>>>>>> for
>>>>>>>>> you if you transfer us the dump (be aware that some objects can
>>>>>>>>> contain
>>>>>>>>> some user credentials, but that really doesn't interest us).
>>>>>>>>>
>>>>>>>>> Tks,
>>>>>>>>> - Eric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> the problem persists, but now I have a new exception in the log:
>>>>>>>>>>
>>>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail
>>>>>>>>>> while
>>>>>>>>>> spooling PermGen space
>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail
>>>>>>>>>> while
>>>>>>>>>> spooling (class: javax/mail/MessagingException, method:
>>>>>>>>>> superToString
>>>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException,
>>>>>>>>>> method:
>>>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant
>>>>>>>>>> pool
>>>>>>>>>> index
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail
>>>>>>>>>> while
>>>>>>>>>> spooling PermGen space
>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Nicola.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>>>>>>>
>>>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1]
>>>>>>>>>>> of 20/5.
>>>>>>>>>>>
>>>>>>>>>>> If you still encounter the OOM, you can further analyse with
>>>>>>>>>>> "eclipse
>>>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>>>>>>>>> objects that eat memory (launch james with standard Mx 512M,
>>>>>>>>>>> otherwise
>>>>>>>>>>> you will need more time to laod dump).
>>>>>>>>>>>
>>>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>>>>>>
>>>>>>>>>>> Tks,
>>>>>>>>>>> - Eric
>>>>>>>>>>>
>>>>>>>>>>> [1] http://s.apache.org/30x
>>>>>>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>>>>>>
>>>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Eric,
>>>>>>>>>>>> I'm using snapshot
>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>>>>>>
>>>>>>>>>>>> now I'll try to install snapshot
>>>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>>>>>>> and I'll add the complete debugging path in the startup script
>>>>>>>>>>>> to
>>>>>>>>>>>> further analyse the dump.
>>>>>>>>>>>>
>>>>>>>>>>>> Many thanks for your help,
>>>>>>>>>>>>
>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Tks for your answer.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>>>>>>> - maildir
>>>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's
>>>>>>>>>>>>> rejected
>>>>>>>>>>>>> via smtp hook)
>>>>>>>>>>>>> - not many users
>>>>>>>>>>>>>
>>>>>>>>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders
>>>>>>>>>>>>> empty).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the
>>>>>>>>>>>>> issue.
>>>>>>>>>>>>> Would it come from recent
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>>>>>>> btw, which version are you using ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> The complete debugging path would be to add
>>>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in
>>>>>>>>>>>>> the
>>>>>>>>>>>>> startup script and to further analyse the dump: it will give
>>>>>>>>>>>>> for sure
>>>>>>>>>>>>> the cause of oom.
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> at this time we don't have very high mail traffic, more or
>>>>>>>>>>>>>> less 500/600
>>>>>>>>>>>>>> mail per day.
>>>>>>>>>>>>>> James is configured as smtp and imap servers and I think that
>>>>>>>>>>>>>> most of
>>>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active
>>>>>>>>>>>>>> clients at the
>>>>>>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846
>>>>>>>>>>>>>> Got<tag>: 1125
>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846
>>>>>>>>>>>>>> Got<command>:
>>>>>>>>>>>>>> noop
>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601
>>>>>>>>>>>>>> executing hook
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process
>>>>>>>>>>>>>> request
>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352
>>>>>>>>>>>>>> Got<tag>: A05974
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage"
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail
>>>>>>>>>>>>>> will be
>>>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/"
>>>>>>>>>>>>>> folder. If
>>>>>>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Last thing: James is installed on a server machine whit
>>>>>>>>>>>>>> following
>>>>>>>>>>>>>> characteristics:
>>>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel
>>>>>>>>>>>>>> 2.6.37.6-0.5
>>>>>>>>>>>>>> java version: 1.6.0_25
>>>>>>>>>>>>>> RAM: 8Gb
>>>>>>>>>>>>>> Architecture: x86_64
>>>>>>>>>>>>>> CPU(s): 6
>>>>>>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>>>>>>> Thread(s) per core: 1
>>>>>>>>>>>>>> Core(s) per socket: 6
>>>>>>>>>>>>>> CPU socket(s): 1
>>>>>>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>>>>>>> CPU family: 16
>>>>>>>>>>>>>> Model: 8
>>>>>>>>>>>>>> Stepping: 1
>>>>>>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for your help,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine
>>>>>>>>>>>>>>>> for one or
>>>>>>>>>>>>>>>> two days until I start having memory problems and I need to
>>>>>>>>>>>>>>>> restart
>>>>>>>>>>>>>>>> James.
>>>>>>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>> WARN 04:46:27,928 |
>>>>>>>>>>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process
>>>>>>>>>>>>>>>> request
>>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to
>>>>>>>>>>>>>>>> restart James.
>>>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m
>>>>>>>>>>>>>>>> -Xmx1024m)
>>>>>>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail:
>>>>>>>>>>>>> server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail:
>>>>>>>>>>> server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Hi Nicola,
Yes, you're right.
The problem arise in fact when the mail must be delivered to/cc local
and non-local recipients, so when LocalDelivery and RemoteDelivery
mailets must come into the play for the same mail.
I see cpu rising to 100% and leaking files in blob transfer - I didn't
wait to see oom, but I'm pretty sure it will come.
I also tested with remotedelivery configured with a gateway and without
a gateway (in my case, non gateway make the mail undeliverable to the
remote recipient). This does not change anything (bug is there for both
configuration).
Could I ask you to open a JIRA
(https://issues.apache.org/jira/browse/JAMES) for this and define it as
blocker.
Tks very much for the follow,
- Eric
On 27/05/2011 15:09, Nicola Salvemini wrote:
> Hello,
> my mailetcontainer.xml is very standard, I don't use custom servlet or
> something strange.
> Could you please try sending only with James without using your ISP SMTP
> server?
>
> Thanks,
>
> Nicola.
>
>
>
>
> On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
>> Replicated issue is the begin of the solution :)
>>
>> I sent a mail with my account in cc, and everything happens fine
>> (delivered, no remaining files in blob-transfer).
>>
>> It goes via my ISP SMTP server before reaching James, but that should
>> not change anything.
>>
>> Did you change mailetcontainer.xml or any other configuration ?
>>
>> Tks,
>> - Eric
>>
>> On 27/05/2011 12:34, Nicola Salvemini wrote:
>>> Hello,
>>>
>>> I have a good news! I'm able to replicate problem: I noticed the problem
>>> appears when James try to delivery email having a local address in CC.
>>> Please consider following example:
>>> James is configured as SMTP server for mydomain.com. If it try to
>>> deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
>>> is the result:
>>>
>>> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
>>> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
>>> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
>>> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
>>>
>>> Nothing more!
>>>
>>>> From this point on, James is not able to send any mail. All emails
>>> remain in "store/activemq/blob-transfer/" until OOM is reached.
>>> To resolve it's necessary delete email in
>>> "store/activemq/blob-transfer/" and restart the server.
>>>
>>> I think the problem is this. What do you think?
>>>
>>> Nicola.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
>>>> OK, let's see what the next dump tells us.
>>>>
>>>> Once a OOM occurs, the consecutive exceptions can come from anywhere and
>>>> should not be considered too much. Memory becomes short, jvm is confused...
>>>>
>>>> Have you still some messages stuck in queue and remaining files in
>>>> "store/activemq/blob-transfer/" ?
>>>>
>>>> I was wondering about your mailetcontainer.xml configuration. Did you
>>>> configure something special? Is it possible that for some reasones, the
>>>> processor route a mails takes is "infinite" ?
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>>
>>>> On 26/05/2011 15:18, Nicola Salvemini wrote:
>>>>> Hi,
>>>>>
>>>>> here new exceptions found in the log. I installed Eclipe Memory Analyzer
>>>>> but I was not able to open the dump file because it was corrupt. I'll
>>>>> retry with next OOM.
>>>>>
>>>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
>>>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
>>>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
>>>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
>>>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
>>>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
>>>>> state root reached
>>>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>> dequeue next message;
>>>>> nested exception is:
>>>>> javax.jms.JMSException: PermGen space
>>>>> at
>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>> at
>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>> at
>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>> at
>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.springframework.jms.connection.CachingConnectionFactory
>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>> at
>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>> ... 2 more
>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>> at java.lang.String.intern(Native Method)
>>>>> at
>>>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
>>>>> at javax.management.ObjectName.construct(ObjectName.java:644)
>>>>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
>>>>> at
>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
>>>>> at
>>>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
>>>>> at
>>>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
>>>>> at
>>>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
>>>>> at
>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
>>>>> at
>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>> at
>>>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
>>>>> at
>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>> at
>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>> at
>>>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>>>> at
>>>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
>>>>> at
>>>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
>>>>> at
>>>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
>>>>> at
>>>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
>>>>> at org.apache.activemq.broker.TransportConnection
>>>>> $1.onCommand(TransportConnection.java:185)
>>>>> at
>>>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>>>> at
>>>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>>>> at
>>>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
>>>>> at
>>>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>>>> at org.apache.activemq.thread.PooledTaskRunner
>>>>> $1.run(PooledTaskRunner.java:43)
>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>>>> at java.util.concurrent.ThreadPoolExecutor
>>>>> $Worker.run(ThreadPoolExecutor.java:908)
>>>>> ... 1 more
>>>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
>>>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
>>>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
>>>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
>>>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
>>>>> abcd@abcd.org to yyyyy@zzzz.net
>>>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
>>>>> state root reached
>>>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
>>>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
>>>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
>>>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
>>>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
>>>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
>>>>> state root reached
>>>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
>>>>> RemoteDelivery.run()
>>>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
>>>>> RemoteDelivery.run()
>>>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
>>>>> RemoteDelivery.run()
>>>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
>>>>> RemoteDelivery.run()
>>>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
>>>>> spooling Unable to dequeue next message
>>>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>>>> dequeue next message;
>>>>> nested exception is:
>>>>> javax.jms.JMSException: PermGen space
>>>>> at
>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>>>> at
>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> Caused by: javax.jms.JMSException: PermGen space
>>>>> at
>>>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>>>> at
>>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>>>> at
>>>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>>>> at
>>>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.springframework.jms.connection.CachingConnectionFactory
>>>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>>>> at $Proxy61.createConsumer(Unknown Source)
>>>>> at
>>>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>>>> ... 2 more
>>>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>>>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>>>>>> If you go via the menu and open the dump file, it will propose you
>>>>>> (after a few minutes anlaysis) a report that will present you
>>>>>> "suspects". You can copy/paste this report on ml.
>>>>>>
>>>>>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
>>>>>> sure it allows to analyze taken dumps, it more for live profiling).
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>>
>>>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
>>>>>>> way to analize dump file. Have you any suggestion?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>>>>>> Hi,
>>>>>>>> Getting OOM can have various side effects, so I bet the
>>>>>>>> java.lang.VerifyError is a result of the OOM.
>>>>>>>>
>>>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>>>>>
>>>>>>>> If you have now time to analyze to find the "suspect", we can do it for
>>>>>>>> you if you transfer us the dump (be aware that some objects can contain
>>>>>>>> some user credentials, but that really doesn't interest us).
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> - Eric
>>>>>>>>
>>>>>>>>
>>>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> the problem persists, but now I have a new exception in the log:
>>>>>>>>>
>>>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
>>>>>>>>> spooling PermGen space
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
>>>>>>>>> spooling (class: javax/mail/MessagingException, method: superToString
>>>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
>>>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
>>>>>>>>> index
>>>>>>>>> at
>>>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>>>>>> at
>>>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>>>>>> at
>>>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
>>>>>>>>> spooling PermGen space
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nicola.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>>>>>>>>>
>>>>>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
>>>>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>>>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
>>>>>>>>>> you will need more time to laod dump).
>>>>>>>>>>
>>>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> - Eric
>>>>>>>>>>
>>>>>>>>>> [1] http://s.apache.org/30x
>>>>>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>>>>>
>>>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>>>>>> Hi Eric,
>>>>>>>>>>> I'm using snapshot
>>>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>>>>>
>>>>>>>>>>> now I'll try to install snapshot
>>>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>>>>>> and I'll add the complete debugging path in the startup script to
>>>>>>>>>>> further analyse the dump.
>>>>>>>>>>>
>>>>>>>>>>> Many thanks for your help,
>>>>>>>>>>>
>>>>>>>>>>> Nicola.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>>>>>> Tks for your answer.
>>>>>>>>>>>>
>>>>>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>>>>>> - maildir
>>>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>>>>>>>>>> via smtp hook)
>>>>>>>>>>>> - not many users
>>>>>>>>>>>>
>>>>>>>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>>>>>>>>>
>>>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>>>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>>>>>> btw, which version are you using ?
>>>>>>>>>>>>
>>>>>>>>>>>> The complete debugging path would be to add
>>>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>>>>>>>>>> startup script and to further analyse the dump: it will give for sure
>>>>>>>>>>>> the cause of oom.
>>>>>>>>>>>>
>>>>>>>>>>>> - Eric
>>>>>>>>>>>>
>>>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>>>>>>>>>> mail per day.
>>>>>>>>>>>>> James is configured as smtp and imap servers and I think that most of
>>>>>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>>>>>
>>>>>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>>>>>
>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>>>>>>>>>> noop
>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Last thing: James is installed on a server machine whit following
>>>>>>>>>>>>> characteristics:
>>>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>>>>>>>>>> java version: 1.6.0_25
>>>>>>>>>>>>> RAM: 8Gb
>>>>>>>>>>>>> Architecture: x86_64
>>>>>>>>>>>>> CPU(s): 6
>>>>>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>>>>>> Thread(s) per core: 1
>>>>>>>>>>>>> Core(s) per socket: 6
>>>>>>>>>>>>> CPU socket(s): 1
>>>>>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>>>>>> CPU family: 16
>>>>>>>>>>>>> Model: 8
>>>>>>>>>>>>> Stepping: 1
>>>>>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for your help,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Tks,
>>>>>>>>>>>>>> - Eric
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>>>>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>>>>>>>>>> James.
>>>>>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and then
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hello,
my mailetcontainer.xml is very standard, I don't use custom servlet or
something strange.
Could you please try sending only with James without using your ISP SMTP
server?
Thanks,
Nicola.
On Fri, 2011-05-27 at 12:42 +0200, Eric Charles wrote:
> Replicated issue is the begin of the solution :)
>
> I sent a mail with my account in cc, and everything happens fine
> (delivered, no remaining files in blob-transfer).
>
> It goes via my ISP SMTP server before reaching James, but that should
> not change anything.
>
> Did you change mailetcontainer.xml or any other configuration ?
>
> Tks,
> - Eric
>
> On 27/05/2011 12:34, Nicola Salvemini wrote:
> > Hello,
> >
> > I have a good news! I'm able to replicate problem: I noticed the problem
> > appears when James try to delivery email having a local address in CC.
> > Please consider following example:
> > James is configured as SMTP server for mydomain.com. If it try to
> > deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
> > is the result:
> >
> > DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
> > Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
> > DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
> > DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
> >
> > Nothing more!
> >
> >> From this point on, James is not able to send any mail. All emails
> > remain in "store/activemq/blob-transfer/" until OOM is reached.
> > To resolve it's necessary delete email in
> > "store/activemq/blob-transfer/" and restart the server.
> >
> > I think the problem is this. What do you think?
> >
> > Nicola.
> >
> >
> >
> >
> >
> >
> > On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
> >> OK, let's see what the next dump tells us.
> >>
> >> Once a OOM occurs, the consecutive exceptions can come from anywhere and
> >> should not be considered too much. Memory becomes short, jvm is confused...
> >>
> >> Have you still some messages stuck in queue and remaining files in
> >> "store/activemq/blob-transfer/" ?
> >>
> >> I was wondering about your mailetcontainer.xml configuration. Did you
> >> configure something special? Is it possible that for some reasones, the
> >> processor route a mails takes is "infinite" ?
> >>
> >> Tks,
> >> - Eric
> >>
> >>
> >> On 26/05/2011 15:18, Nicola Salvemini wrote:
> >>> Hi,
> >>>
> >>> here new exceptions found in the log. I installed Eclipe Memory Analyzer
> >>> but I was not able to open the dump file because it was corrupt. I'll
> >>> retry with next OOM.
> >>>
> >>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
> >>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
> >>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
> >>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
> >>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
> >>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> >>> xxxxx@xxxxx.com to yyyyy@zzzz.net
> >>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
> >>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> >>> xxxxx@xxxxx.com to hhhhh@zzzz.net
> >>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
> >>> state root reached
> >>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> >>> dequeue next message;
> >>> nested exception is:
> >>> javax.jms.JMSException: PermGen space
> >>> at
> >>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> >>> at
> >>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> >>> at java.lang.Thread.run(Thread.java:662)
> >>> Caused by: javax.jms.JMSException: PermGen space
> >>> at
> >>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> >>> at
> >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> >>> at
> >>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> >>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> >>> at
> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>> at org.springframework.jms.connection.CachingConnectionFactory
> >>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> >>> at $Proxy61.createConsumer(Unknown Source)
> >>> at
> >>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> >>> ... 2 more
> >>> Caused by: java.lang.OutOfMemoryError: PermGen space
> >>> at java.lang.String.intern(Native Method)
> >>> at
> >>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
> >>> at javax.management.ObjectName.construct(ObjectName.java:644)
> >>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
> >>> at
> >>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
> >>> at
> >>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
> >>> at
> >>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
> >>> at
> >>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
> >>> at
> >>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
> >>> at
> >>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>> at
> >>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
> >>> at
> >>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>> at
> >>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>> at
> >>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> >>> at
> >>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
> >>> at
> >>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
> >>> at
> >>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
> >>> at
> >>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> >>> at org.apache.activemq.broker.TransportConnection
> >>> $1.onCommand(TransportConnection.java:185)
> >>> at
> >>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> >>> at
> >>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> >>> at
> >>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
> >>> at
> >>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> >>> at org.apache.activemq.thread.PooledTaskRunner
> >>> $1.run(PooledTaskRunner.java:43)
> >>> at java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.runTask(ThreadPoolExecutor.java:886)
> >>> at java.util.concurrent.ThreadPoolExecutor
> >>> $Worker.run(ThreadPoolExecutor.java:908)
> >>> ... 1 more
> >>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
> >>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
> >>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
> >>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
> >>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
> >>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
> >>> abcd@abcd.org to yyyyy@zzzz.net
> >>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
> >>> state root reached
> >>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
> >>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
> >>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
> >>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
> >>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
> >>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
> >>> xxxxx@xxxxx.com to yyyyy@zzzz.net
> >>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
> >>> state root reached
> >>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
> >>> RemoteDelivery.run()
> >>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
> >>> RemoteDelivery.run()
> >>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
> >>> RemoteDelivery.run()
> >>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
> >>> RemoteDelivery.run()
> >>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
> >>> spooling Unable to dequeue next message
> >>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> >>> dequeue next message;
> >>> nested exception is:
> >>> javax.jms.JMSException: PermGen space
> >>> at
> >>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> >>> at
> >>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> >>> at java.lang.Thread.run(Thread.java:662)
> >>> Caused by: javax.jms.JMSException: PermGen space
> >>> at
> >>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> >>> at
> >>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> >>> at
> >>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> >>> at
> >>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> >>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> >>> at
> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>> at java.lang.reflect.Method.invoke(Method.java:597)
> >>> at org.springframework.jms.connection.CachingConnectionFactory
> >>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> >>> at $Proxy61.createConsumer(Unknown Source)
> >>> at
> >>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> >>> ... 2 more
> >>> Caused by: java.lang.OutOfMemoryError: PermGen space
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
> >>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
> >>>> If you go via the menu and open the dump file, it will propose you
> >>>> (after a few minutes anlaysis) a report that will present you
> >>>> "suspects". You can copy/paste this report on ml.
> >>>>
> >>>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
> >>>> sure it allows to analyze taken dumps, it more for live profiling).
> >>>>
> >>>> Tks,
> >>>> - Eric
> >>>>
> >>>>
> >>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
> >>>>> Hi Eric,
> >>>>>
> >>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
> >>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
> >>>>> way to analize dump file. Have you any suggestion?
> >>>>>
> >>>>> Regards,
> >>>>> Nicola.
> >>>>>
> >>>>>
> >>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
> >>>>>> Hi,
> >>>>>> Getting OOM can have various side effects, so I bet the
> >>>>>> java.lang.VerifyError is a result of the OOM.
> >>>>>>
> >>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
> >>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
> >>>>>>
> >>>>>> If you have now time to analyze to find the "suspect", we can do it for
> >>>>>> you if you transfer us the dump (be aware that some objects can contain
> >>>>>> some user credentials, but that really doesn't interest us).
> >>>>>>
> >>>>>> Tks,
> >>>>>> - Eric
> >>>>>>
> >>>>>>
> >>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> the problem persists, but now I have a new exception in the log:
> >>>>>>>
> >>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> >>>>>>> spooling PermGen space
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> >>>>>>> spooling (class: javax/mail/MessagingException, method: superToString
> >>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
> >>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> >>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
> >>>>>>> index
> >>>>>>> at
> >>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> >>>>>>> at
> >>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> >>>>>>> at
> >>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> >>>>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> >>>>>>> spooling PermGen space
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>
> >>>>>>>
> >>>>>>> Nicola.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> >>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
> >>>>>>>>
> >>>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
> >>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
> >>>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
> >>>>>>>> you will need more time to laod dump).
> >>>>>>>>
> >>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
> >>>>>>>> james-server-context.xml (see changes from [1]).
> >>>>>>>>
> >>>>>>>> Tks,
> >>>>>>>> - Eric
> >>>>>>>>
> >>>>>>>> [1] http://s.apache.org/30x
> >>>>>>>> [2] http://www.eclipse.org/mat/
> >>>>>>>>
> >>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
> >>>>>>>>> Hi Eric,
> >>>>>>>>> I'm using snapshot
> >>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
> >>>>>>>>>
> >>>>>>>>> now I'll try to install snapshot
> >>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
> >>>>>>>>> and I'll add the complete debugging path in the startup script to
> >>>>>>>>> further analyse the dump.
> >>>>>>>>>
> >>>>>>>>> Many thanks for your help,
> >>>>>>>>>
> >>>>>>>>> Nicola.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >>>>>>>>>> Tks for your answer.
> >>>>>>>>>>
> >>>>>>>>>> So you've got more or less the same configuration as I have:
> >>>>>>>>>> - maildir
> >>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >>>>>>>>>> via smtp hook)
> >>>>>>>>>> - not many users
> >>>>>>>>>>
> >>>>>>>>>> My config here is really stable after weeks (no oom, low memory
> >>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
> >>>>>>>>>>
> >>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
> >>>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >>>>>>>>>> btw, which version are you using ?
> >>>>>>>>>>
> >>>>>>>>>> The complete debugging path would be to add
> >>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >>>>>>>>>> startup script and to further analyse the dump: it will give for sure
> >>>>>>>>>> the cause of oom.
> >>>>>>>>>>
> >>>>>>>>>> - Eric
> >>>>>>>>>>
> >>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
> >>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
> >>>>>>>>>>> mail per day.
> >>>>>>>>>>> James is configured as smtp and imap servers and I think that most of
> >>>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
> >>>>>>>>>>> same time making requests to the James IMAP server).
> >>>>>>>>>>>
> >>>>>>>>>>>> This will allow us to better define the place to look at.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, it is in most cases.
> >>>>>>>>>>>
> >>>>>>>>>>>> Is the log still showing messages after the OOM?
> >>>>>>>>>>>
> >>>>>>>>>>> Yes, please consider following log sequence:
> >>>>>>>>>>>
> >>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>>>>>>>>>> noop
> >>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
> >>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>>>>>>>>>> the number of mail rises too high then the problem occurs.
> >>>>>>>>>>>
> >>>>>>>>>>> Last thing: James is installed on a server machine whit following
> >>>>>>>>>>> characteristics:
> >>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>>>>>>>>>> java version: 1.6.0_25
> >>>>>>>>>>> RAM: 8Gb
> >>>>>>>>>>> Architecture: x86_64
> >>>>>>>>>>> CPU(s): 6
> >>>>>>>>>>> On-line CPU(s) list: 0-5
> >>>>>>>>>>> Thread(s) per core: 1
> >>>>>>>>>>> Core(s) per socket: 6
> >>>>>>>>>>> CPU socket(s): 1
> >>>>>>>>>>> Vendor ID: AuthenticAMD
> >>>>>>>>>>> CPU family: 16
> >>>>>>>>>>> Model: 8
> >>>>>>>>>>> Stepping: 1
> >>>>>>>>>>> CPU MHz: 2599.966
> >>>>>>>>>>> BogoMIPS: 5200.12
> >>>>>>>>>>> Virtualization: AMD-V
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for your help,
> >>>>>>>>>>>
> >>>>>>>>>>> Nicola.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Tks,
> >>>>>>>>>>>> - Eric
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>>>>>>>>>> two days until I start having memory problems and I need to restart
> >>>>>>>>>>>>> James.
> >>>>>>>>>>>>> The problems begin with a long series of messages such as
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> and then
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>>>>>>>>>> but the problem keeps coming back.
> >>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
> >>>>>>>>>>>>> Thanks in advance,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Nicola.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Replicated issue is the begin of the solution :)
I sent a mail with my account in cc, and everything happens fine
(delivered, no remaining files in blob-transfer).
It goes via my ISP SMTP server before reaching James, but that should
not change anything.
Did you change mailetcontainer.xml or any other configuration ?
Tks,
- Eric
On 27/05/2011 12:34, Nicola Salvemini wrote:
> Hello,
>
> I have a good news! I'm able to replicate problem: I noticed the problem
> appears when James try to delivery email having a local address in CC.
> Please consider following example:
> James is configured as SMTP server for mydomain.com. If it try to
> deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
> is the result:
>
> DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
> Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
> DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
> DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
>
> Nothing more!
>
>> From this point on, James is not able to send any mail. All emails
> remain in "store/activemq/blob-transfer/" until OOM is reached.
> To resolve it's necessary delete email in
> "store/activemq/blob-transfer/" and restart the server.
>
> I think the problem is this. What do you think?
>
> Nicola.
>
>
>
>
>
>
> On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
>> OK, let's see what the next dump tells us.
>>
>> Once a OOM occurs, the consecutive exceptions can come from anywhere and
>> should not be considered too much. Memory becomes short, jvm is confused...
>>
>> Have you still some messages stuck in queue and remaining files in
>> "store/activemq/blob-transfer/" ?
>>
>> I was wondering about your mailetcontainer.xml configuration. Did you
>> configure something special? Is it possible that for some reasones, the
>> processor route a mails takes is "infinite" ?
>>
>> Tks,
>> - Eric
>>
>>
>> On 26/05/2011 15:18, Nicola Salvemini wrote:
>>> Hi,
>>>
>>> here new exceptions found in the log. I installed Eclipe Memory Analyzer
>>> but I was not able to open the dump file because it was corrupt. I'll
>>> retry with next OOM.
>>>
>>> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
>>> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
>>> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
>>> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
>>> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
>>> xxxxx@xxxxx.com to hhhhh@zzzz.net
>>> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
>>> state root reached
>>> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>> dequeue next message;
>>> nested exception is:
>>> javax.jms.JMSException: PermGen space
>>> at
>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>> at
>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: javax.jms.JMSException: PermGen space
>>> at
>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>> at
>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>> at
>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>> at
>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.springframework.jms.connection.CachingConnectionFactory
>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>> at $Proxy61.createConsumer(Unknown Source)
>>> at
>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>> ... 2 more
>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>> at java.lang.String.intern(Native Method)
>>> at
>>> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
>>> at javax.management.ObjectName.construct(ObjectName.java:644)
>>> at javax.management.ObjectName.<init>(ObjectName.java:1403)
>>> at
>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
>>> at
>>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
>>> at
>>> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
>>> at
>>> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
>>> at
>>> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
>>> at
>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>> at
>>> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
>>> at
>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>> at
>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>> at
>>> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
>>> at
>>> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
>>> at
>>> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
>>> at
>>> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
>>> at
>>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
>>> at org.apache.activemq.broker.TransportConnection
>>> $1.onCommand(TransportConnection.java:185)
>>> at
>>> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
>>> at
>>> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
>>> at
>>> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
>>> at
>>> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>>> at org.apache.activemq.thread.PooledTaskRunner
>>> $1.run(PooledTaskRunner.java:43)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.runTask(ThreadPoolExecutor.java:886)
>>> at java.util.concurrent.ThreadPoolExecutor
>>> $Worker.run(ThreadPoolExecutor.java:908)
>>> ... 1 more
>>> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
>>> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
>>> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
>>> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
>>> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
>>> abcd@abcd.org to yyyyy@zzzz.net
>>> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
>>> state root reached
>>> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
>>> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
>>> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
>>> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
>>> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
>>> xxxxx@xxxxx.com to yyyyy@zzzz.net
>>> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
>>> state root reached
>>> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
>>> RemoteDelivery.run()
>>> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
>>> RemoteDelivery.run()
>>> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
>>> RemoteDelivery.run()
>>> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
>>> RemoteDelivery.run()
>>> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
>>> spooling Unable to dequeue next message
>>> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
>>> dequeue next message;
>>> nested exception is:
>>> javax.jms.JMSException: PermGen space
>>> at
>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
>>> at
>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
>>> at java.lang.Thread.run(Thread.java:662)
>>> Caused by: javax.jms.JMSException: PermGen space
>>> at
>>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
>>> at
>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
>>> at
>>> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
>>> at
>>> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
>>> at
>>> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
>>> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.springframework.jms.connection.CachingConnectionFactory
>>> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
>>> at $Proxy61.createConsumer(Unknown Source)
>>> at
>>> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
>>> ... 2 more
>>> Caused by: java.lang.OutOfMemoryError: PermGen space
>>>
>>>
>>>
>>>
>>> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>>>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>>>> If you go via the menu and open the dump file, it will propose you
>>>> (after a few minutes anlaysis) a report that will present you
>>>> "suspects". You can copy/paste this report on ml.
>>>>
>>>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
>>>> sure it allows to analyze taken dumps, it more for live profiling).
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>>
>>>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>>>> Hi Eric,
>>>>>
>>>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
>>>>> way to analize dump file. Have you any suggestion?
>>>>>
>>>>> Regards,
>>>>> Nicola.
>>>>>
>>>>>
>>>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>>>> Hi,
>>>>>> Getting OOM can have various side effects, so I bet the
>>>>>> java.lang.VerifyError is a result of the OOM.
>>>>>>
>>>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>>>
>>>>>> If you have now time to analyze to find the "suspect", we can do it for
>>>>>> you if you transfer us the dump (be aware that some objects can contain
>>>>>> some user credentials, but that really doesn't interest us).
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>>
>>>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> the problem persists, but now I have a new exception in the log:
>>>>>>>
>>>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
>>>>>>> spooling PermGen space
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
>>>>>>> spooling (class: javax/mail/MessagingException, method: superToString
>>>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
>>>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
>>>>>>> index
>>>>>>> at
>>>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>>>> at
>>>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>>>> at
>>>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
>>>>>>> spooling PermGen space
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>>>>>>>
>>>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
>>>>>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
>>>>>>>> you will need more time to laod dump).
>>>>>>>>
>>>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> - Eric
>>>>>>>>
>>>>>>>> [1] http://s.apache.org/30x
>>>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>>>
>>>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>>>> Hi Eric,
>>>>>>>>> I'm using snapshot
>>>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>>>
>>>>>>>>> now I'll try to install snapshot
>>>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>>>> and I'll add the complete debugging path in the startup script to
>>>>>>>>> further analyse the dump.
>>>>>>>>>
>>>>>>>>> Many thanks for your help,
>>>>>>>>>
>>>>>>>>> Nicola.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>>>> Tks for your answer.
>>>>>>>>>>
>>>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>>>> - maildir
>>>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>>>>>>>> via smtp hook)
>>>>>>>>>> - not many users
>>>>>>>>>>
>>>>>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>>>>>>>
>>>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>>>> btw, which version are you using ?
>>>>>>>>>>
>>>>>>>>>> The complete debugging path would be to add
>>>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>>>>>>>> startup script and to further analyse the dump: it will give for sure
>>>>>>>>>> the cause of oom.
>>>>>>>>>>
>>>>>>>>>> - Eric
>>>>>>>>>>
>>>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>>>>>>>> mail per day.
>>>>>>>>>>> James is configured as smtp and imap servers and I think that most of
>>>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>>>
>>>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>>>
>>>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>>>>>
>>>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>>>
>>>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>>>
>>>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>>>
>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>>>>>>>> noop
>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>>>
>>>>>>>>>>> Last thing: James is installed on a server machine whit following
>>>>>>>>>>> characteristics:
>>>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>>>>>>>> java version: 1.6.0_25
>>>>>>>>>>> RAM: 8Gb
>>>>>>>>>>> Architecture: x86_64
>>>>>>>>>>> CPU(s): 6
>>>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>>>> Thread(s) per core: 1
>>>>>>>>>>> Core(s) per socket: 6
>>>>>>>>>>> CPU socket(s): 1
>>>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>>>> CPU family: 16
>>>>>>>>>>> Model: 8
>>>>>>>>>>> Stepping: 1
>>>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your help,
>>>>>>>>>>>
>>>>>>>>>>> Nicola.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Tks,
>>>>>>>>>>>> - Eric
>>>>>>>>>>>>
>>>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>>>>>>>> James.
>>>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>>>
>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>>>
>>>>>>>>>>>>> and then
>>>>>>>>>>>>>
>>>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>>>
>>>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Nicola.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hello,
I have a good news! I'm able to replicate problem: I noticed the problem
appears when James try to delivery email having a local address in CC.
Please consider following example:
James is configured as SMTP server for mydomain.com. If it try to
deliver email to foo@gmail.com having myaddress@mydomain.com in CC, this
is the result:
DEBUG 12:01:43,641 | james.mailspooler | ==== Begin processing mail
Mail1306490503638-8a087915-aded-4c10-9b37-be8d543652c1====
DEBUG 12:01:43,641 | james.mailprocessor | Call MailProcessor root
DEBUG 12:01:43,773 | james.mailprocessor | Call MailProcessor transport
Nothing more!
>From this point on, James is not able to send any mail. All emails
remain in "store/activemq/blob-transfer/" until OOM is reached.
To resolve it's necessary delete email in
"store/activemq/blob-transfer/" and restart the server.
I think the problem is this. What do you think?
Nicola.
On Thu, 2011-05-26 at 15:28 +0200, Eric Charles wrote:
> OK, let's see what the next dump tells us.
>
> Once a OOM occurs, the consecutive exceptions can come from anywhere and
> should not be considered too much. Memory becomes short, jvm is confused...
>
> Have you still some messages stuck in queue and remaining files in
> "store/activemq/blob-transfer/" ?
>
> I was wondering about your mailetcontainer.xml configuration. Did you
> configure something special? Is it possible that for some reasones, the
> processor route a mails takes is "infinite" ?
>
> Tks,
> - Eric
>
>
> On 26/05/2011 15:18, Nicola Salvemini wrote:
> > Hi,
> >
> > here new exceptions found in the log. I installed Eclipe Memory Analyzer
> > but I was not able to open the dump file because it was corrupt. I'll
> > retry with next OOM.
> >
> > DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
> > Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
> > DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
> > DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
> > INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
> > Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> > xxxxx@xxxxx.com to yyyyy@zzzz.net
> > INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
> > Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> > xxxxx@xxxxx.com to hhhhh@zzzz.net
> > DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
> > state root reached
> > ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> > dequeue next message;
> > nested exception is:
> > javax.jms.JMSException: PermGen space
> > at
> > org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> > at
> > org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: javax.jms.JMSException: PermGen space
> > at
> > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> > at
> > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> > at
> > org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> > at
> > org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.springframework.jms.connection.CachingConnectionFactory
> > $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> > at $Proxy61.createConsumer(Unknown Source)
> > at
> > org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> > ... 2 more
> > Caused by: java.lang.OutOfMemoryError: PermGen space
> > at java.lang.String.intern(Native Method)
> > at
> > javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
> > at javax.management.ObjectName.construct(ObjectName.java:644)
> > at javax.management.ObjectName.<init>(ObjectName.java:1403)
> > at
> > org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
> > at
> > org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
> > at
> > org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
> > at
> > org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
> > at
> > org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
> > at
> > org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> > at
> > org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
> > at
> > org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> > at
> > org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> > at
> > org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> > at
> > org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
> > at
> > org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
> > at
> > org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
> > at
> > org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> > at org.apache.activemq.broker.TransportConnection
> > $1.onCommand(TransportConnection.java:185)
> > at
> > org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> > at
> > org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> > at
> > org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
> > at
> > org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> > at org.apache.activemq.thread.PooledTaskRunner
> > $1.run(PooledTaskRunner.java:43)
> > at java.util.concurrent.ThreadPoolExecutor
> > $Worker.runTask(ThreadPoolExecutor.java:886)
> > at java.util.concurrent.ThreadPoolExecutor
> > $Worker.run(ThreadPoolExecutor.java:908)
> > ... 1 more
> > DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
> > Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
> > DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
> > DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
> > INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
> > Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
> > abcd@abcd.org to yyyyy@zzzz.net
> > DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
> > state root reached
> > DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
> > Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
> > DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
> > DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
> > INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
> > Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
> > xxxxx@xxxxx.com to yyyyy@zzzz.net
> > DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
> > state root reached
> > ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > INFO 12:27:26,135 | james.mailetcontext | Exception caught in
> > RemoteDelivery.run()
> > INFO 12:27:38,052 | james.mailetcontext | Exception caught in
> > RemoteDelivery.run()
> > INFO 12:27:38,724 | james.mailetcontext | Exception caught in
> > RemoteDelivery.run()
> > ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > INFO 12:28:42,830 | james.mailetcontext | Exception caught in
> > RemoteDelivery.run()
> > ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
> > spooling Unable to dequeue next message
> > org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> > dequeue next message;
> > nested exception is:
> > javax.jms.JMSException: PermGen space
> > at
> > org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> > at
> > org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: javax.jms.JMSException: PermGen space
> > at
> > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> > at
> > org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> > at
> > org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> > at
> > org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> > at
> > org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.springframework.jms.connection.CachingConnectionFactory
> > $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> > at $Proxy61.createConsumer(Unknown Source)
> > at
> > org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> > ... 2 more
> > Caused by: java.lang.OutOfMemoryError: PermGen space
> >
> >
> >
> >
> > On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
> >> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
> >> If you go via the menu and open the dump file, it will propose you
> >> (after a few minutes anlaysis) a report that will present you
> >> "suspects". You can copy/paste this report on ml.
> >>
> >> There's also YourKit (http://www.yourkit.com/) but I never used it (not
> >> sure it allows to analyze taken dumps, it more for live profiling).
> >>
> >> Tks,
> >> - Eric
> >>
> >>
> >> On 26/05/2011 11:46, Nicola Salvemini wrote:
> >>> Hi Eric,
> >>>
> >>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
> >>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
> >>> way to analize dump file. Have you any suggestion?
> >>>
> >>> Regards,
> >>> Nicola.
> >>>
> >>>
> >>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
> >>>> Hi,
> >>>> Getting OOM can have various side effects, so I bet the
> >>>> java.lang.VerifyError is a result of the OOM.
> >>>>
> >>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
> >>>> -XX:HeapDumpPath=/your_path" and got a dump?
> >>>>
> >>>> If you have now time to analyze to find the "suspect", we can do it for
> >>>> you if you transfer us the dump (be aware that some objects can contain
> >>>> some user credentials, but that really doesn't interest us).
> >>>>
> >>>> Tks,
> >>>> - Eric
> >>>>
> >>>>
> >>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
> >>>>> Hello,
> >>>>>
> >>>>> the problem persists, but now I have a new exception in the log:
> >>>>>
> >>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> >>>>> spooling PermGen space
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> >>>>> spooling (class: javax/mail/MessagingException, method: superToString
> >>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
> >>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> >>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
> >>>>> index
> >>>>> at
> >>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> >>>>> at
> >>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> >>>>> at
> >>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> >>>>> at java.lang.Thread.run(Thread.java:662)
> >>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> >>>>> spooling PermGen space
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>
> >>>>>
> >>>>> Nicola.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> >>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
> >>>>>>
> >>>>>> If you still encounter the OOM, you can further analyse with "eclipse
> >>>>>> mat" [2] for example and ask for "suspects". It will show you the
> >>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
> >>>>>> you will need more time to laod dump).
> >>>>>>
> >>>>>> Optionally, you can try to re-add the prefetchPolicy in
> >>>>>> james-server-context.xml (see changes from [1]).
> >>>>>>
> >>>>>> Tks,
> >>>>>> - Eric
> >>>>>>
> >>>>>> [1] http://s.apache.org/30x
> >>>>>> [2] http://www.eclipse.org/mat/
> >>>>>>
> >>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
> >>>>>>> Hi Eric,
> >>>>>>> I'm using snapshot
> >>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
> >>>>>>>
> >>>>>>> now I'll try to install snapshot
> >>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
> >>>>>>> and I'll add the complete debugging path in the startup script to
> >>>>>>> further analyse the dump.
> >>>>>>>
> >>>>>>> Many thanks for your help,
> >>>>>>>
> >>>>>>> Nicola.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >>>>>>>> Tks for your answer.
> >>>>>>>>
> >>>>>>>> So you've got more or less the same configuration as I have:
> >>>>>>>> - maildir
> >>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >>>>>>>> via smtp hook)
> >>>>>>>> - not many users
> >>>>>>>>
> >>>>>>>> My config here is really stable after weeks (no oom, low memory
> >>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
> >>>>>>>>
> >>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
> >>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >>>>>>>> btw, which version are you using ?
> >>>>>>>>
> >>>>>>>> The complete debugging path would be to add
> >>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >>>>>>>> startup script and to further analyse the dump: it will give for sure
> >>>>>>>> the cause of oom.
> >>>>>>>>
> >>>>>>>> - Eric
> >>>>>>>>
> >>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>>>>>>>
> >>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>>>>>>>> Hi,
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Can you give us a rough estimate of the traffic you have?
> >>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
> >>>>>>>>> mail per day.
> >>>>>>>>> James is configured as smtp and imap servers and I think that most of
> >>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
> >>>>>>>>> same time making requests to the James IMAP server).
> >>>>>>>>>
> >>>>>>>>>> This will allow us to better define the place to look at.
> >>>>>>>>>>
> >>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
> >>>>>>>>>
> >>>>>>>>> Yes, it is in most cases.
> >>>>>>>>>
> >>>>>>>>>> Is the log still showing messages after the OOM?
> >>>>>>>>>
> >>>>>>>>> Yes, please consider following log sequence:
> >>>>>>>>>
> >>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>>>>>>>> noop
> >>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
> >>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>>>>>>>> the number of mail rises too high then the problem occurs.
> >>>>>>>>>
> >>>>>>>>> Last thing: James is installed on a server machine whit following
> >>>>>>>>> characteristics:
> >>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>>>>>>>> java version: 1.6.0_25
> >>>>>>>>> RAM: 8Gb
> >>>>>>>>> Architecture: x86_64
> >>>>>>>>> CPU(s): 6
> >>>>>>>>> On-line CPU(s) list: 0-5
> >>>>>>>>> Thread(s) per core: 1
> >>>>>>>>> Core(s) per socket: 6
> >>>>>>>>> CPU socket(s): 1
> >>>>>>>>> Vendor ID: AuthenticAMD
> >>>>>>>>> CPU family: 16
> >>>>>>>>> Model: 8
> >>>>>>>>> Stepping: 1
> >>>>>>>>> CPU MHz: 2599.966
> >>>>>>>>> BogoMIPS: 5200.12
> >>>>>>>>> Virtualization: AMD-V
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Thanks for your help,
> >>>>>>>>>
> >>>>>>>>> Nicola.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Tks,
> >>>>>>>>>> - Eric
> >>>>>>>>>>
> >>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hello,
> >>>>>>>>>>>
> >>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>>>>>>>> two days until I start having memory problems and I need to restart
> >>>>>>>>>>> James.
> >>>>>>>>>>> The problems begin with a long series of messages such as
> >>>>>>>>>>>
> >>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>>>> | Failed to register MBean:
> >>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>>>
> >>>>>>>>>>> and then
> >>>>>>>>>>>
> >>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>>>
> >>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>>>>>>>> but the problem keeps coming back.
> >>>>>>>>>>> Does anyone have a suggestion to solve the problem?
> >>>>>>>>>>> Thanks in advance,
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> Nicola.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
OK, let's see what the next dump tells us.
Once a OOM occurs, the consecutive exceptions can come from anywhere and
should not be considered too much. Memory becomes short, jvm is confused...
Have you still some messages stuck in queue and remaining files in
"store/activemq/blob-transfer/" ?
I was wondering about your mailetcontainer.xml configuration. Did you
configure something special? Is it possible that for some reasones, the
processor route a mails takes is "infinite" ?
Tks,
- Eric
On 26/05/2011 15:18, Nicola Salvemini wrote:
> Hi,
>
> here new exceptions found in the log. I installed Eclipe Memory Analyzer
> but I was not able to open the dump file because it was corrupt. I'll
> retry with next OOM.
>
> DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
> DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
> DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
> INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> xxxxx@xxxxx.com to yyyyy@zzzz.net
> INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
> Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
> xxxxx@xxxxx.com to hhhhh@zzzz.net
> DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
> state root reached
> ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> dequeue next message;
> nested exception is:
> javax.jms.JMSException: PermGen space
> at
> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> at
> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.jms.JMSException: PermGen space
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> at
> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> at
> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.jms.connection.CachingConnectionFactory
> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> at $Proxy61.createConsumer(Unknown Source)
> at
> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> ... 2 more
> Caused by: java.lang.OutOfMemoryError: PermGen space
> at java.lang.String.intern(Native Method)
> at
> javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
> at javax.management.ObjectName.construct(ObjectName.java:644)
> at javax.management.ObjectName.<init>(ObjectName.java:1403)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
> at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
> at
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
> at
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
> at
> org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
> at
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
> at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
> at org.apache.activemq.broker.TransportConnection
> $1.onCommand(TransportConnection.java:185)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
> at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
> at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
> at org.apache.activemq.thread.PooledTaskRunner
> $1.run(PooledTaskRunner.java:43)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> ... 1 more
> DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
> DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
> DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
> INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
> Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
> abcd@abcd.org to yyyyy@zzzz.net
> DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
> state root reached
> DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
> DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
> DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
> INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
> Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
> xxxxx@xxxxx.com to yyyyy@zzzz.net
> DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
> state root reached
> ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> INFO 12:27:26,135 | james.mailetcontext | Exception caught in
> RemoteDelivery.run()
> INFO 12:27:38,052 | james.mailetcontext | Exception caught in
> RemoteDelivery.run()
> INFO 12:27:38,724 | james.mailetcontext | Exception caught in
> RemoteDelivery.run()
> ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> INFO 12:28:42,830 | james.mailetcontext | Exception caught in
> RemoteDelivery.run()
> ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
> spooling Unable to dequeue next message
> org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
> dequeue next message;
> nested exception is:
> javax.jms.JMSException: PermGen space
> at
> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
> at
> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: javax.jms.JMSException: PermGen space
> at
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
> at
> org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
> at
> org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
> at
> org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
> at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.springframework.jms.connection.CachingConnectionFactory
> $CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
> at $Proxy61.createConsumer(Unknown Source)
> at
> org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
> ... 2 more
> Caused by: java.lang.OutOfMemoryError: PermGen space
>
>
>
>
> On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
>> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
>> If you go via the menu and open the dump file, it will propose you
>> (after a few minutes anlaysis) a report that will present you
>> "suspects". You can copy/paste this report on ml.
>>
>> There's also YourKit (http://www.yourkit.com/) but I never used it (not
>> sure it allows to analyze taken dumps, it more for live profiling).
>>
>> Tks,
>> - Eric
>>
>>
>> On 26/05/2011 11:46, Nicola Salvemini wrote:
>>> Hi Eric,
>>>
>>> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
>>> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
>>> way to analize dump file. Have you any suggestion?
>>>
>>> Regards,
>>> Nicola.
>>>
>>>
>>> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>>>> Hi,
>>>> Getting OOM can have various side effects, so I bet the
>>>> java.lang.VerifyError is a result of the OOM.
>>>>
>>>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>>>> -XX:HeapDumpPath=/your_path" and got a dump?
>>>>
>>>> If you have now time to analyze to find the "suspect", we can do it for
>>>> you if you transfer us the dump (be aware that some objects can contain
>>>> some user credentials, but that really doesn't interest us).
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>>
>>>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>>>> Hello,
>>>>>
>>>>> the problem persists, but now I have a new exception in the log:
>>>>>
>>>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
>>>>> spooling PermGen space
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
>>>>> spooling (class: javax/mail/MessagingException, method: superToString
>>>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
>>>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
>>>>> index
>>>>> at
>>>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>>>> at
>>>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>>>> at
>>>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>>>> at java.lang.Thread.run(Thread.java:662)
>>>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
>>>>> spooling PermGen space
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>>>>>
>>>>>> If you still encounter the OOM, you can further analyse with "eclipse
>>>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
>>>>>> you will need more time to laod dump).
>>>>>>
>>>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>>>> james-server-context.xml (see changes from [1]).
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>> [1] http://s.apache.org/30x
>>>>>> [2] http://www.eclipse.org/mat/
>>>>>>
>>>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>>>> Hi Eric,
>>>>>>> I'm using snapshot
>>>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>>>
>>>>>>> now I'll try to install snapshot
>>>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>>>> and I'll add the complete debugging path in the startup script to
>>>>>>> further analyse the dump.
>>>>>>>
>>>>>>> Many thanks for your help,
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>>>> Tks for your answer.
>>>>>>>>
>>>>>>>> So you've got more or less the same configuration as I have:
>>>>>>>> - maildir
>>>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>>>>>> via smtp hook)
>>>>>>>> - not many users
>>>>>>>>
>>>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>>>>>
>>>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>>>> btw, which version are you using ?
>>>>>>>>
>>>>>>>> The complete debugging path would be to add
>>>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>>>>>> startup script and to further analyse the dump: it will give for sure
>>>>>>>> the cause of oom.
>>>>>>>>
>>>>>>>> - Eric
>>>>>>>>
>>>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>>>
>>>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>>>> Hi,
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>>>>>> mail per day.
>>>>>>>>> James is configured as smtp and imap servers and I think that most of
>>>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>>>>>> same time making requests to the James IMAP server).
>>>>>>>>>
>>>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>>>
>>>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>>>
>>>>>>>>> Yes, it is in most cases.
>>>>>>>>>
>>>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>>>
>>>>>>>>> Yes, please consider following log sequence:
>>>>>>>>>
>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>>>>>> noop
>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>>>
>>>>>>>>> Last thing: James is installed on a server machine whit following
>>>>>>>>> characteristics:
>>>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>>>>>> java version: 1.6.0_25
>>>>>>>>> RAM: 8Gb
>>>>>>>>> Architecture: x86_64
>>>>>>>>> CPU(s): 6
>>>>>>>>> On-line CPU(s) list: 0-5
>>>>>>>>> Thread(s) per core: 1
>>>>>>>>> Core(s) per socket: 6
>>>>>>>>> CPU socket(s): 1
>>>>>>>>> Vendor ID: AuthenticAMD
>>>>>>>>> CPU family: 16
>>>>>>>>> Model: 8
>>>>>>>>> Stepping: 1
>>>>>>>>> CPU MHz: 2599.966
>>>>>>>>> BogoMIPS: 5200.12
>>>>>>>>> Virtualization: AMD-V
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks for your help,
>>>>>>>>>
>>>>>>>>> Nicola.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Tks,
>>>>>>>>>> - Eric
>>>>>>>>>>
>>>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>>>>>> James.
>>>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>>>
>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>>>> | Failed to register MBean:
>>>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>>>
>>>>>>>>>>> and then
>>>>>>>>>>>
>>>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>>>
>>>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>>>>>> but the problem keeps coming back.
>>>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>>>> Thanks in advance,
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Nicola.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hi,
here new exceptions found in the log. I installed Eclipe Memory Analyzer
but I was not able to open the dump file because it was corrupt. I'll
retry with next OOM.
DEBUG 12:21:12,022 | james.mailspooler | ==== Begin processing mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92====
DEBUG 12:21:12,023 | james.mailprocessor | Call MailProcessor root
DEBUG 12:21:12,753 | james.mailprocessor | Call MailProcessor transport
INFO 12:21:18,821 | james.mailetcontext | Local delivered mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
xxxxx@xxxxx.com to yyyyy@zzzz.net
INFO 12:21:20,157 | james.mailetcontext | Local delivered mail
Mail1306405272019-60eec5f7-1b95-49aa-b269-8116f29fda92 sucessfully from
xxxxx@xxxxx.com to hhhhh@zzzz.net
DEBUG 12:21:20,158 | james.mailprocessor | End of mailetprocessor for
state root reached
ERROR 12:21:56,674 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
dequeue next message;
nested exception is:
javax.jms.JMSException: PermGen space
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.JMSException: PermGen space
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jms.connection.CachingConnectionFactory
$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
at $Proxy61.createConsumer(Unknown Source)
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
... 2 more
Caused by: java.lang.OutOfMemoryError: PermGen space
at java.lang.String.intern(Native Method)
at
javax.management.ObjectName.setCanonicalName(ObjectName.java:816)
at javax.management.ObjectName.construct(ObjectName.java:644)
at javax.management.ObjectName.<init>(ObjectName.java:1403)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.registerSubscription(ManagedRegionBroker.java:184)
at
org.apache.activemq.broker.jmx.ManagedQueueRegion.createSubscription(ManagedQueueRegion.java:45)
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:268)
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:444)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.addConsumer(ManagedRegionBroker.java:240)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:91)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:89)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:550)
at
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:349)
at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
at org.apache.activemq.broker.TransportConnection
$1.onCommand(TransportConnection.java:185)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:219)
at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
at org.apache.activemq.thread.PooledTaskRunner
$1.run(PooledTaskRunner.java:43)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
DEBUG 12:22:06,560 | james.mailspooler | ==== Begin processing mail
Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1====
DEBUG 12:22:06,560 | james.mailprocessor | Call MailProcessor root
DEBUG 12:22:06,591 | james.mailprocessor | Call MailProcessor transport
INFO 12:22:16,207 | james.mailetcontext | Local delivered mail
Mail1306405326556-44ac5713-6208-44a7-985c-86782b4a7db1 sucessfully from
abcd@abcd.org to yyyyy@zzzz.net
DEBUG 12:22:16,208 | james.mailprocessor | End of mailetprocessor for
state root reached
DEBUG 12:22:22,101 | james.mailspooler | ==== Begin processing mail
Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc====
DEBUG 12:22:22,101 | james.mailprocessor | Call MailProcessor root
DEBUG 12:22:22,130 | james.mailprocessor | Call MailProcessor transport
INFO 12:22:22,873 | james.mailetcontext | Local delivered mail
Mail1306405342098-6ecf752a-5ec3-4bef-aa54-07b94cb7bbbc sucessfully from
xxxxx@xxxxx.com to yyyyy@zzzz.net
DEBUG 12:22:22,874 | james.mailprocessor | End of mailetprocessor for
state root reached
ERROR 12:25:41,320 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
INFO 12:27:26,135 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
INFO 12:27:38,052 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
INFO 12:27:38,724 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
ERROR 12:28:16,597 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:30,033 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:32,152 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:42,077 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
INFO 12:28:42,830 | james.mailetcontext | Exception caught in
RemoteDelivery.run()
ERROR 12:28:44,283 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
ERROR 12:28:47,108 | james.mailspooler | Exception processing mail while
spooling Unable to dequeue next message
org.apache.james.queue.api.MailQueue$MailQueueException: Unable to
dequeue next message;
nested exception is:
javax.jms.JMSException: PermGen space
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:171)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:135)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.jms.JMSException: PermGen space
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1287)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1874)
at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:254)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1116)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:1060)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:973)
at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.springframework.jms.connection.CachingConnectionFactory
$CachedSessionInvocationHandler.invoke(CachingConnectionFactory.java:344)
at $Proxy61.createConsumer(Unknown Source)
at
org.apache.james.queue.jms.JMSMailQueue.deQueue(JMSMailQueue.java:109)
... 2 more
Caused by: java.lang.OutOfMemoryError: PermGen space
On Thu, 2011-05-26 at 11:55 +0200, Eric Charles wrote:
> I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
> If you go via the menu and open the dump file, it will propose you
> (after a few minutes anlaysis) a report that will present you
> "suspects". You can copy/paste this report on ml.
>
> There's also YourKit (http://www.yourkit.com/) but I never used it (not
> sure it allows to analyze taken dumps, it more for live profiling).
>
> Tks,
> - Eric
>
>
> On 26/05/2011 11:46, Nicola Salvemini wrote:
> > Hi Eric,
> >
> > yes, I added "-XX:+HeapDumpOnOutOfMemoryError
> > -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
> > way to analize dump file. Have you any suggestion?
> >
> > Regards,
> > Nicola.
> >
> >
> > On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
> >> Hi,
> >> Getting OOM can have various side effects, so I bet the
> >> java.lang.VerifyError is a result of the OOM.
> >>
> >> Did you add "-XX:+HeapDumpOnOutOfMemoryError
> >> -XX:HeapDumpPath=/your_path" and got a dump?
> >>
> >> If you have now time to analyze to find the "suspect", we can do it for
> >> you if you transfer us the dump (be aware that some objects can contain
> >> some user credentials, but that really doesn't interest us).
> >>
> >> Tks,
> >> - Eric
> >>
> >>
> >> On 26/05/2011 09:20, Nicola Salvemini wrote:
> >>> Hello,
> >>>
> >>> the problem persists, but now I have a new exception in the log:
> >>>
> >>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> >>> spooling PermGen space
> >>> java.lang.OutOfMemoryError: PermGen space
> >>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> >>> spooling (class: javax/mail/MessagingException, method: superToString
> >>> signature: ()Ljava/lang/String;) Illegal constant pool index
> >>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> >>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
> >>> index
> >>> at
> >>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> >>> at
> >>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> >>> at
> >>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> >>> at java.lang.Thread.run(Thread.java:662)
> >>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> >>> spooling PermGen space
> >>> java.lang.OutOfMemoryError: PermGen space
> >>>
> >>>
> >>> Nicola.
> >>>
> >>>
> >>>
> >>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> >>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
> >>>>
> >>>> If you still encounter the OOM, you can further analyse with "eclipse
> >>>> mat" [2] for example and ask for "suspects". It will show you the
> >>>> objects that eat memory (launch james with standard Mx 512M, otherwise
> >>>> you will need more time to laod dump).
> >>>>
> >>>> Optionally, you can try to re-add the prefetchPolicy in
> >>>> james-server-context.xml (see changes from [1]).
> >>>>
> >>>> Tks,
> >>>> - Eric
> >>>>
> >>>> [1] http://s.apache.org/30x
> >>>> [2] http://www.eclipse.org/mat/
> >>>>
> >>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
> >>>>> Hi Eric,
> >>>>> I'm using snapshot
> >>>>> james-server-container-spring-3.0-M3-20110523.043027-400
> >>>>>
> >>>>> now I'll try to install snapshot
> >>>>> james-server-container-spring-3.0-M3-20110525.115759-407
> >>>>> and I'll add the complete debugging path in the startup script to
> >>>>> further analyse the dump.
> >>>>>
> >>>>> Many thanks for your help,
> >>>>>
> >>>>> Nicola.
> >>>>>
> >>>>>
> >>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >>>>>> Tks for your answer.
> >>>>>>
> >>>>>> So you've got more or less the same configuration as I have:
> >>>>>> - maildir
> >>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >>>>>> via smtp hook)
> >>>>>> - not many users
> >>>>>>
> >>>>>> My config here is really stable after weeks (no oom, low memory
> >>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
> >>>>>>
> >>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
> >>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >>>>>> btw, which version are you using ?
> >>>>>>
> >>>>>> The complete debugging path would be to add
> >>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >>>>>> startup script and to further analyse the dump: it will give for sure
> >>>>>> the cause of oom.
> >>>>>>
> >>>>>> - Eric
> >>>>>>
> >>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>>>>>
> >>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>>>>>> Hi,
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Can you give us a rough estimate of the traffic you have?
> >>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>>>>>
> >>>>>>>
> >>>>>>> at this time we don't have very high mail traffic, more or less 500/600
> >>>>>>> mail per day.
> >>>>>>> James is configured as smtp and imap servers and I think that most of
> >>>>>>> the traffic is IMAP (currently there are about ten active clients at the
> >>>>>>> same time making requests to the James IMAP server).
> >>>>>>>
> >>>>>>>> This will allow us to better define the place to look at.
> >>>>>>>>
> >>>>>>>> Also, is the CPU always remain 100% when you have that issue?
> >>>>>>>
> >>>>>>> Yes, it is in most cases.
> >>>>>>>
> >>>>>>>> Is the log still showing messages after the OOM?
> >>>>>>>
> >>>>>>> Yes, please consider following log sequence:
> >>>>>>>
> >>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>>>>>> noop
> >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>>>>>
> >>>>>>>
> >>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
> >>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>>>>>> the number of mail rises too high then the problem occurs.
> >>>>>>>
> >>>>>>> Last thing: James is installed on a server machine whit following
> >>>>>>> characteristics:
> >>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>>>>>> java version: 1.6.0_25
> >>>>>>> RAM: 8Gb
> >>>>>>> Architecture: x86_64
> >>>>>>> CPU(s): 6
> >>>>>>> On-line CPU(s) list: 0-5
> >>>>>>> Thread(s) per core: 1
> >>>>>>> Core(s) per socket: 6
> >>>>>>> CPU socket(s): 1
> >>>>>>> Vendor ID: AuthenticAMD
> >>>>>>> CPU family: 16
> >>>>>>> Model: 8
> >>>>>>> Stepping: 1
> >>>>>>> CPU MHz: 2599.966
> >>>>>>> BogoMIPS: 5200.12
> >>>>>>> Virtualization: AMD-V
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks for your help,
> >>>>>>>
> >>>>>>> Nicola.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Tks,
> >>>>>>>> - Eric
> >>>>>>>>
> >>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>>>>>
> >>>>>>>>> Hello,
> >>>>>>>>>
> >>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>>>>>> two days until I start having memory problems and I need to restart
> >>>>>>>>> James.
> >>>>>>>>> The problems begin with a long series of messages such as
> >>>>>>>>>
> >>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>> | Failed to register MBean:
> >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>> | Failed to register MBean:
> >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>> | Failed to register MBean:
> >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>>>> | Failed to register MBean:
> >>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>>>
> >>>>>>>>> and then
> >>>>>>>>>
> >>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>>>
> >>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>>>>>> but the problem keeps coming back.
> >>>>>>>>> Does anyone have a suggestion to solve the problem?
> >>>>>>>>> Thanks in advance,
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Nicola.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> ---------------------------------------------------------------------
> >>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
I'm using Eclipe Memory Analyzer (http://www.eclipse.org/mat/).
If you go via the menu and open the dump file, it will propose you
(after a few minutes anlaysis) a report that will present you
"suspects". You can copy/paste this report on ml.
There's also YourKit (http://www.yourkit.com/) but I never used it (not
sure it allows to analyze taken dumps, it more for live profiling).
Tks,
- Eric
On 26/05/2011 11:46, Nicola Salvemini wrote:
> Hi Eric,
>
> yes, I added "-XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
> way to analize dump file. Have you any suggestion?
>
> Regards,
> Nicola.
>
>
> On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
>> Hi,
>> Getting OOM can have various side effects, so I bet the
>> java.lang.VerifyError is a result of the OOM.
>>
>> Did you add "-XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=/your_path" and got a dump?
>>
>> If you have now time to analyze to find the "suspect", we can do it for
>> you if you transfer us the dump (be aware that some objects can contain
>> some user credentials, but that really doesn't interest us).
>>
>> Tks,
>> - Eric
>>
>>
>> On 26/05/2011 09:20, Nicola Salvemini wrote:
>>> Hello,
>>>
>>> the problem persists, but now I have a new exception in the log:
>>>
>>> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
>>> spooling PermGen space
>>> java.lang.OutOfMemoryError: PermGen space
>>> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
>>> spooling (class: javax/mail/MessagingException, method: superToString
>>> signature: ()Ljava/lang/String;) Illegal constant pool index
>>> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
>>> superToString signature: ()Ljava/lang/String;) Illegal constant pool
>>> index
>>> at
>>> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
>>> at
>>> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
>>> at
>>> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
>>> at java.lang.Thread.run(Thread.java:662)
>>> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
>>> spooling PermGen space
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>>
>>> Nicola.
>>>
>>>
>>>
>>> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>>>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>>>
>>>> If you still encounter the OOM, you can further analyse with "eclipse
>>>> mat" [2] for example and ask for "suspects". It will show you the
>>>> objects that eat memory (launch james with standard Mx 512M, otherwise
>>>> you will need more time to laod dump).
>>>>
>>>> Optionally, you can try to re-add the prefetchPolicy in
>>>> james-server-context.xml (see changes from [1]).
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>> [1] http://s.apache.org/30x
>>>> [2] http://www.eclipse.org/mat/
>>>>
>>>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>>>> Hi Eric,
>>>>> I'm using snapshot
>>>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>>>
>>>>> now I'll try to install snapshot
>>>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>>>> and I'll add the complete debugging path in the startup script to
>>>>> further analyse the dump.
>>>>>
>>>>> Many thanks for your help,
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>>>> Tks for your answer.
>>>>>>
>>>>>> So you've got more or less the same configuration as I have:
>>>>>> - maildir
>>>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>>>> via smtp hook)
>>>>>> - not many users
>>>>>>
>>>>>> My config here is really stable after weeks (no oom, low memory
>>>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>>>
>>>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>>>> btw, which version are you using ?
>>>>>>
>>>>>> The complete debugging path would be to add
>>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>>>> startup script and to further analyse the dump: it will give for sure
>>>>>> the cause of oom.
>>>>>>
>>>>>> - Eric
>>>>>>
>>>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>>>
>>>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>>>> Hi,
>>>>>>> Hi,
>>>>>>>
>>>>>>>>
>>>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>>>
>>>>>>>
>>>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>>>> mail per day.
>>>>>>> James is configured as smtp and imap servers and I think that most of
>>>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>>>> same time making requests to the James IMAP server).
>>>>>>>
>>>>>>>> This will allow us to better define the place to look at.
>>>>>>>>
>>>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>>>
>>>>>>> Yes, it is in most cases.
>>>>>>>
>>>>>>>> Is the log still showing messages after the OOM?
>>>>>>>
>>>>>>> Yes, please consider following log sequence:
>>>>>>>
>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>>>> noop
>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>>>
>>>>>>>
>>>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>>>> the number of mail rises too high then the problem occurs.
>>>>>>>
>>>>>>> Last thing: James is installed on a server machine whit following
>>>>>>> characteristics:
>>>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>>>> java version: 1.6.0_25
>>>>>>> RAM: 8Gb
>>>>>>> Architecture: x86_64
>>>>>>> CPU(s): 6
>>>>>>> On-line CPU(s) list: 0-5
>>>>>>> Thread(s) per core: 1
>>>>>>> Core(s) per socket: 6
>>>>>>> CPU socket(s): 1
>>>>>>> Vendor ID: AuthenticAMD
>>>>>>> CPU family: 16
>>>>>>> Model: 8
>>>>>>> Stepping: 1
>>>>>>> CPU MHz: 2599.966
>>>>>>> BogoMIPS: 5200.12
>>>>>>> Virtualization: AMD-V
>>>>>>>
>>>>>>>
>>>>>>> Thanks for your help,
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Tks,
>>>>>>>> - Eric
>>>>>>>>
>>>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>>>> James.
>>>>>>>>> The problems begin with a long series of messages such as
>>>>>>>>>
>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>> | Failed to register MBean:
>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>> | Failed to register MBean:
>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>> | Failed to register MBean:
>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>>>> | Failed to register MBean:
>>>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>>>
>>>>>>>>> and then
>>>>>>>>>
>>>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>>>
>>>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>>>> but the problem keeps coming back.
>>>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>>>> Thanks in advance,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Nicola.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hi Eric,
yes, I added "-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/your_path" to the command line, but I'm finding a easy
way to analize dump file. Have you any suggestion?
Regards,
Nicola.
On Thu, 2011-05-26 at 11:25 +0200, Eric Charles wrote:
> Hi,
> Getting OOM can have various side effects, so I bet the
> java.lang.VerifyError is a result of the OOM.
>
> Did you add "-XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/your_path" and got a dump?
>
> If you have now time to analyze to find the "suspect", we can do it for
> you if you transfer us the dump (be aware that some objects can contain
> some user credentials, but that really doesn't interest us).
>
> Tks,
> - Eric
>
>
> On 26/05/2011 09:20, Nicola Salvemini wrote:
> > Hello,
> >
> > the problem persists, but now I have a new exception in the log:
> >
> > ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> > spooling PermGen space
> > java.lang.OutOfMemoryError: PermGen space
> > ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> > spooling (class: javax/mail/MessagingException, method: superToString
> > signature: ()Ljava/lang/String;) Illegal constant pool index
> > java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> > superToString signature: ()Ljava/lang/String;) Illegal constant pool
> > index
> > at
> > org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> > at
> > org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> > at
> > org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> > at java.lang.Thread.run(Thread.java:662)
> > ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> > spooling PermGen space
> > java.lang.OutOfMemoryError: PermGen space
> >
> >
> > Nicola.
> >
> >
> >
> > On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> >> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
> >>
> >> If you still encounter the OOM, you can further analyse with "eclipse
> >> mat" [2] for example and ask for "suspects". It will show you the
> >> objects that eat memory (launch james with standard Mx 512M, otherwise
> >> you will need more time to laod dump).
> >>
> >> Optionally, you can try to re-add the prefetchPolicy in
> >> james-server-context.xml (see changes from [1]).
> >>
> >> Tks,
> >> - Eric
> >>
> >> [1] http://s.apache.org/30x
> >> [2] http://www.eclipse.org/mat/
> >>
> >> On 25/05/2011 15:56, Nicola Salvemini wrote:
> >>> Hi Eric,
> >>> I'm using snapshot
> >>> james-server-container-spring-3.0-M3-20110523.043027-400
> >>>
> >>> now I'll try to install snapshot
> >>> james-server-container-spring-3.0-M3-20110525.115759-407
> >>> and I'll add the complete debugging path in the startup script to
> >>> further analyse the dump.
> >>>
> >>> Many thanks for your help,
> >>>
> >>> Nicola.
> >>>
> >>>
> >>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >>>> Tks for your answer.
> >>>>
> >>>> So you've got more or less the same configuration as I have:
> >>>> - maildir
> >>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >>>> via smtp hook)
> >>>> - not many users
> >>>>
> >>>> My config here is really stable after weeks (no oom, low memory
> >>>> consumption when i monitor via JMX, blob-transfer folders empty).
> >>>>
> >>>> Yes, it sounds like some mails remains blocked, giving the issue.
> >>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >>>> btw, which version are you using ?
> >>>>
> >>>> The complete debugging path would be to add
> >>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >>>> startup script and to further analyse the dump: it will give for sure
> >>>> the cause of oom.
> >>>>
> >>>> - Eric
> >>>>
> >>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>>>
> >>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>>>> Hi,
> >>>>> Hi,
> >>>>>
> >>>>>>
> >>>>>> Can you give us a rough estimate of the traffic you have?
> >>>>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>>>
> >>>>>
> >>>>> at this time we don't have very high mail traffic, more or less 500/600
> >>>>> mail per day.
> >>>>> James is configured as smtp and imap servers and I think that most of
> >>>>> the traffic is IMAP (currently there are about ten active clients at the
> >>>>> same time making requests to the James IMAP server).
> >>>>>
> >>>>>> This will allow us to better define the place to look at.
> >>>>>>
> >>>>>> Also, is the CPU always remain 100% when you have that issue?
> >>>>>
> >>>>> Yes, it is in most cases.
> >>>>>
> >>>>>> Is the log still showing messages after the OOM?
> >>>>>
> >>>>> Yes, please consider following log sequence:
> >>>>>
> >>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>>>> noop
> >>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>>>
> >>>>>
> >>>>> I noticed that sometimes the problem occurs for a "blockage" of
> >>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>>>> the number of mail rises too high then the problem occurs.
> >>>>>
> >>>>> Last thing: James is installed on a server machine whit following
> >>>>> characteristics:
> >>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>>>> java version: 1.6.0_25
> >>>>> RAM: 8Gb
> >>>>> Architecture: x86_64
> >>>>> CPU(s): 6
> >>>>> On-line CPU(s) list: 0-5
> >>>>> Thread(s) per core: 1
> >>>>> Core(s) per socket: 6
> >>>>> CPU socket(s): 1
> >>>>> Vendor ID: AuthenticAMD
> >>>>> CPU family: 16
> >>>>> Model: 8
> >>>>> Stepping: 1
> >>>>> CPU MHz: 2599.966
> >>>>> BogoMIPS: 5200.12
> >>>>> Virtualization: AMD-V
> >>>>>
> >>>>>
> >>>>> Thanks for your help,
> >>>>>
> >>>>> Nicola.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>>>> Tks,
> >>>>>> - Eric
> >>>>>>
> >>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>>>> two days until I start having memory problems and I need to restart
> >>>>>>> James.
> >>>>>>> The problems begin with a long series of messages such as
> >>>>>>>
> >>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>> | Failed to register MBean:
> >>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>> | Failed to register MBean:
> >>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>> | Failed to register MBean:
> >>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>>>> | Failed to register MBean:
> >>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>>>
> >>>>>>> and then
> >>>>>>>
> >>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>>>
> >>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>>>> but the problem keeps coming back.
> >>>>>>> Does anyone have a suggestion to solve the problem?
> >>>>>>> Thanks in advance,
> >>>>>>>
> >>>>>>>
> >>>>>>> Nicola.
> >>>>>>>
> >>>>>>>
> >>>>>>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ---------------------------------------------------------------------
> >>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Hi,
Getting OOM can have various side effects, so I bet the
java.lang.VerifyError is a result of the OOM.
Did you add "-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/your_path" and got a dump?
If you have now time to analyze to find the "suspect", we can do it for
you if you transfer us the dump (be aware that some objects can contain
some user credentials, but that really doesn't interest us).
Tks,
- Eric
On 26/05/2011 09:20, Nicola Salvemini wrote:
> Hello,
>
> the problem persists, but now I have a new exception in the log:
>
> ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
> spooling PermGen space
> java.lang.OutOfMemoryError: PermGen space
> ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
> spooling (class: javax/mail/MessagingException, method: superToString
> signature: ()Ljava/lang/String;) Illegal constant pool index
> java.lang.VerifyError: (class: javax/mail/MessagingException, method:
> superToString signature: ()Ljava/lang/String;) Illegal constant pool
> index
> at
> org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
> at
> org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
> at
> org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
> at java.lang.Thread.run(Thread.java:662)
> ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
> spooling PermGen space
> java.lang.OutOfMemoryError: PermGen space
>
>
> Nicola.
>
>
>
> On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>
>> If you still encounter the OOM, you can further analyse with "eclipse
>> mat" [2] for example and ask for "suspects". It will show you the
>> objects that eat memory (launch james with standard Mx 512M, otherwise
>> you will need more time to laod dump).
>>
>> Optionally, you can try to re-add the prefetchPolicy in
>> james-server-context.xml (see changes from [1]).
>>
>> Tks,
>> - Eric
>>
>> [1] http://s.apache.org/30x
>> [2] http://www.eclipse.org/mat/
>>
>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>> Hi Eric,
>>> I'm using snapshot
>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>
>>> now I'll try to install snapshot
>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>> and I'll add the complete debugging path in the startup script to
>>> further analyse the dump.
>>>
>>> Many thanks for your help,
>>>
>>> Nicola.
>>>
>>>
>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>> Tks for your answer.
>>>>
>>>> So you've got more or less the same configuration as I have:
>>>> - maildir
>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>> via smtp hook)
>>>> - not many users
>>>>
>>>> My config here is really stable after weeks (no oom, low memory
>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>
>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>> btw, which version are you using ?
>>>>
>>>> The complete debugging path would be to add
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>> startup script and to further analyse the dump: it will give for sure
>>>> the cause of oom.
>>>>
>>>> - Eric
>>>>
>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>
>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>> Hi,
>>>>> Hi,
>>>>>
>>>>>>
>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>
>>>>>
>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>> mail per day.
>>>>> James is configured as smtp and imap servers and I think that most of
>>>>> the traffic is IMAP (currently there are about ten active clients at the
>>>>> same time making requests to the James IMAP server).
>>>>>
>>>>>> This will allow us to better define the place to look at.
>>>>>>
>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>
>>>>> Yes, it is in most cases.
>>>>>
>>>>>> Is the log still showing messages after the OOM?
>>>>>
>>>>> Yes, please consider following log sequence:
>>>>>
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>> noop
>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>
>>>>>
>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>> the number of mail rises too high then the problem occurs.
>>>>>
>>>>> Last thing: James is installed on a server machine whit following
>>>>> characteristics:
>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>> java version: 1.6.0_25
>>>>> RAM: 8Gb
>>>>> Architecture: x86_64
>>>>> CPU(s): 6
>>>>> On-line CPU(s) list: 0-5
>>>>> Thread(s) per core: 1
>>>>> Core(s) per socket: 6
>>>>> CPU socket(s): 1
>>>>> Vendor ID: AuthenticAMD
>>>>> CPU family: 16
>>>>> Model: 8
>>>>> Stepping: 1
>>>>> CPU MHz: 2599.966
>>>>> BogoMIPS: 5200.12
>>>>> Virtualization: AMD-V
>>>>>
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>> James.
>>>>>>> The problems begin with a long series of messages such as
>>>>>>>
>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> and then
>>>>>>>
>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>
>>>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>>>> but the problem keeps coming back.
>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hello,
the problem persists, but now I have a new exception in the log:
ERROR 09:12:38,851 | james.mailspooler | Exception processing mail while
spooling PermGen space
java.lang.OutOfMemoryError: PermGen space
ERROR 09:12:49,757 | james.mailspooler | Exception processing mail while
spooling (class: javax/mail/MessagingException, method: superToString
signature: ()Ljava/lang/String;) Illegal constant pool index
java.lang.VerifyError: (class: javax/mail/MessagingException, method:
superToString signature: ()Ljava/lang/String;) Illegal constant pool
index
at
org.apache.james.mailetcontainer.camel.CamelMailetProcessor.service(CamelMailetProcessor.java:68)
at
org.apache.james.mailetcontainer.lib.AbstractStateCompositeProcessor.service(AbstractStateCompositeProcessor.java:104)
at
org.apache.james.mailetcontainer.lib.JamesMailSpooler.run(JamesMailSpooler.java:147)
at java.lang.Thread.run(Thread.java:662)
ERROR 09:13:21,452 | james.mailspooler | Exception processing mail while
spooling PermGen space
java.lang.OutOfMemoryError: PermGen space
Nicola.
On Wed, 2011-05-25 at 16:09 +0200, Eric Charles wrote:
> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>
> If you still encounter the OOM, you can further analyse with "eclipse
> mat" [2] for example and ask for "suspects". It will show you the
> objects that eat memory (launch james with standard Mx 512M, otherwise
> you will need more time to laod dump).
>
> Optionally, you can try to re-add the prefetchPolicy in
> james-server-context.xml (see changes from [1]).
>
> Tks,
> - Eric
>
> [1] http://s.apache.org/30x
> [2] http://www.eclipse.org/mat/
>
> On 25/05/2011 15:56, Nicola Salvemini wrote:
> > Hi Eric,
> > I'm using snapshot
> > james-server-container-spring-3.0-M3-20110523.043027-400
> >
> > now I'll try to install snapshot
> > james-server-container-spring-3.0-M3-20110525.115759-407
> > and I'll add the complete debugging path in the startup script to
> > further analyse the dump.
> >
> > Many thanks for your help,
> >
> > Nicola.
> >
> >
> > On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> >> Tks for your answer.
> >>
> >> So you've got more or less the same configuration as I have:
> >> - maildir
> >> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> >> via smtp hook)
> >> - not many users
> >>
> >> My config here is really stable after weeks (no oom, low memory
> >> consumption when i monitor via JMX, blob-transfer folders empty).
> >>
> >> Yes, it sounds like some mails remains blocked, giving the issue.
> >> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> >> btw, which version are you using ?
> >>
> >> The complete debugging path would be to add
> >> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> >> startup script and to further analyse the dump: it will give for sure
> >> the cause of oom.
> >>
> >> - Eric
> >>
> >> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >>>
> >>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >>>> Hi,
> >>> Hi,
> >>>
> >>>>
> >>>> Can you give us a rough estimate of the traffic you have?
> >>>> Mainly SMTP? How many incoming mails/requests per second?
> >>>>
> >>>
> >>> at this time we don't have very high mail traffic, more or less 500/600
> >>> mail per day.
> >>> James is configured as smtp and imap servers and I think that most of
> >>> the traffic is IMAP (currently there are about ten active clients at the
> >>> same time making requests to the James IMAP server).
> >>>
> >>>> This will allow us to better define the place to look at.
> >>>>
> >>>> Also, is the CPU always remain 100% when you have that issue?
> >>>
> >>> Yes, it is in most cases.
> >>>
> >>>> Is the log still showing messages after the OOM?
> >>>
> >>> Yes, please consider following log sequence:
> >>>
> >>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>> java.lang.OutOfMemoryError: PermGen space
> >>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> >>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> >>> noop
> >>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> >>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> >>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>> java.lang.OutOfMemoryError: PermGen space
> >>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> >>> java.lang.OutOfMemoryError: PermGen space
> >>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> >>> java.lang.OutOfMemoryError: PermGen space
> >>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >>>
> >>>
> >>> I noticed that sometimes the problem occurs for a "blockage" of
> >>> ActiveMQ. For some reason it happens that not all spooled mail will be
> >>> processed and they remain in "store/activemq/blob-transfer/" folder. If
> >>> the number of mail rises too high then the problem occurs.
> >>>
> >>> Last thing: James is installed on a server machine whit following
> >>> characteristics:
> >>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> >>> java version: 1.6.0_25
> >>> RAM: 8Gb
> >>> Architecture: x86_64
> >>> CPU(s): 6
> >>> On-line CPU(s) list: 0-5
> >>> Thread(s) per core: 1
> >>> Core(s) per socket: 6
> >>> CPU socket(s): 1
> >>> Vendor ID: AuthenticAMD
> >>> CPU family: 16
> >>> Model: 8
> >>> Stepping: 1
> >>> CPU MHz: 2599.966
> >>> BogoMIPS: 5200.12
> >>> Virtualization: AMD-V
> >>>
> >>>
> >>> Thanks for your help,
> >>>
> >>> Nicola.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>>
> >>>> Tks,
> >>>> - Eric
> >>>>
> >>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>>>> two days until I start having memory problems and I need to restart
> >>>>> James.
> >>>>> The problems begin with a long series of messages such as
> >>>>>
> >>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>> | Failed to register MBean:
> >>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>> | Failed to register MBean:
> >>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>> | Failed to register MBean:
> >>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>>>> | Failed to register MBean:
> >>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>>>
> >>>>> and then
> >>>>>
> >>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>>>> java.lang.OutOfMemoryError: PermGen space
> >>>>>
> >>>>> The CPU usage is over 100% and my only alternative is to restart James.
> >>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>>>> but the problem keeps coming back.
> >>>>> Does anyone have a suggestion to solve the problem?
> >>>>> Thanks in advance,
> >>>>>
> >>>>>
> >>>>> Nicola.
> >>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>>
> >>>>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>>
> >>>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Norman Maurer <no...@googlemail.com>.
Removing the prefetch config will not do anything good...
Bye
Norman
2011/5/25, Eric Charles <er...@apache.org>:
> Sorry, I was meaning "remove the prefetch policy".
>
> For mails blocked in queues, you can read more on
> http://markmail.org/message/v4zxfkbkjjllwrbb and try get more
> information from the jmx console for those messages and on the activemq
> mbeans.
>
> Tks,
> - Eric
>
> On 25/05/2011 16:09, Eric Charles wrote:
>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>
>> If you still encounter the OOM, you can further analyse with "eclipse
>> mat" [2] for example and ask for "suspects". It will show you the
>> objects that eat memory (launch james with standard Mx 512M, otherwise
>> you will need more time to laod dump).
>>
>> Optionally, you can try to re-add the prefetchPolicy in
>> james-server-context.xml (see changes from [1]).
>>
>> Tks,
>> - Eric
>>
>> [1] http://s.apache.org/30x
>> [2] http://www.eclipse.org/mat/
>>
>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>> Hi Eric,
>>> I'm using snapshot
>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>
>>> now I'll try to install snapshot
>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>> and I'll add the complete debugging path in the startup script to
>>> further analyse the dump.
>>>
>>> Many thanks for your help,
>>>
>>> Nicola.
>>>
>>>
>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>> Tks for your answer.
>>>>
>>>> So you've got more or less the same configuration as I have:
>>>> - maildir
>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>> via smtp hook)
>>>> - not many users
>>>>
>>>> My config here is really stable after weeks (no oom, low memory
>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>
>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>> Would it come from recent
>>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>> btw, which version are you using ?
>>>>
>>>> The complete debugging path would be to add
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>> startup script and to further analyse the dump: it will give for sure
>>>> the cause of oom.
>>>>
>>>> - Eric
>>>>
>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>
>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>> Hi,
>>>>> Hi,
>>>>>
>>>>>>
>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>
>>>>>
>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>> mail per day.
>>>>> James is configured as smtp and imap servers and I think that most of
>>>>> the traffic is IMAP (currently there are about ten active clients at
>>>>> the
>>>>> same time making requests to the James IMAP server).
>>>>>
>>>>>> This will allow us to better define the place to look at.
>>>>>>
>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>
>>>>> Yes, it is in most cases.
>>>>>
>>>>>> Is the log still showing messages after the OOM?
>>>>>
>>>>> Yes, please consider following log sequence:
>>>>>
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>> noop
>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>
>>>>>
>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>> the number of mail rises too high then the problem occurs.
>>>>>
>>>>> Last thing: James is installed on a server machine whit following
>>>>> characteristics:
>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>> java version: 1.6.0_25
>>>>> RAM: 8Gb
>>>>> Architecture: x86_64
>>>>> CPU(s): 6
>>>>> On-line CPU(s) list: 0-5
>>>>> Thread(s) per core: 1
>>>>> Core(s) per socket: 6
>>>>> CPU socket(s): 1
>>>>> Vendor ID: AuthenticAMD
>>>>> CPU family: 16
>>>>> Model: 8
>>>>> Stepping: 1
>>>>> CPU MHz: 2599.966
>>>>> BogoMIPS: 5200.12
>>>>> Virtualization: AMD-V
>>>>>
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>> James.
>>>>>>> The problems begin with a long series of messages such as
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>>
>>>>>>> and then
>>>>>>>
>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>
>>>>>>> The CPU usage is over 100% and my only alternative is to restart
>>>>>>> James.
>>>>>>> I tried to increase VM memory (at this time I have -Xms512m
>>>>>>> -Xmx1024m)
>>>>>>> but the problem keeps coming back.
>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Sorry, I was meaning "remove the prefetch policy".
For mails blocked in queues, you can read more on
http://markmail.org/message/v4zxfkbkjjllwrbb and try get more
information from the jmx console for those messages and on the activemq
mbeans.
Tks,
- Eric
On 25/05/2011 16:09, Eric Charles wrote:
> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>
> If you still encounter the OOM, you can further analyse with "eclipse
> mat" [2] for example and ask for "suspects". It will show you the
> objects that eat memory (launch james with standard Mx 512M, otherwise
> you will need more time to laod dump).
>
> Optionally, you can try to re-add the prefetchPolicy in
> james-server-context.xml (see changes from [1]).
>
> Tks,
> - Eric
>
> [1] http://s.apache.org/30x
> [2] http://www.eclipse.org/mat/
>
> On 25/05/2011 15:56, Nicola Salvemini wrote:
>> Hi Eric,
>> I'm using snapshot
>> james-server-container-spring-3.0-M3-20110523.043027-400
>>
>> now I'll try to install snapshot
>> james-server-container-spring-3.0-M3-20110525.115759-407
>> and I'll add the complete debugging path in the startup script to
>> further analyse the dump.
>>
>> Many thanks for your help,
>>
>> Nicola.
>>
>>
>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>> Tks for your answer.
>>>
>>> So you've got more or less the same configuration as I have:
>>> - maildir
>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>> via smtp hook)
>>> - not many users
>>>
>>> My config here is really stable after weeks (no oom, low memory
>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>
>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>> Would it come from recent
>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>> btw, which version are you using ?
>>>
>>> The complete debugging path would be to add
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>> startup script and to further analyse the dump: it will give for sure
>>> the cause of oom.
>>>
>>> - Eric
>>>
>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>
>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>> Hi,
>>>> Hi,
>>>>
>>>>>
>>>>> Can you give us a rough estimate of the traffic you have?
>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>
>>>>
>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>> mail per day.
>>>> James is configured as smtp and imap servers and I think that most of
>>>> the traffic is IMAP (currently there are about ten active clients at
>>>> the
>>>> same time making requests to the James IMAP server).
>>>>
>>>>> This will allow us to better define the place to look at.
>>>>>
>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>
>>>> Yes, it is in most cases.
>>>>
>>>>> Is the log still showing messages after the OOM?
>>>>
>>>> Yes, please consider following log sequence:
>>>>
>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>> noop
>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>
>>>>
>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>> the number of mail rises too high then the problem occurs.
>>>>
>>>> Last thing: James is installed on a server machine whit following
>>>> characteristics:
>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>> java version: 1.6.0_25
>>>> RAM: 8Gb
>>>> Architecture: x86_64
>>>> CPU(s): 6
>>>> On-line CPU(s) list: 0-5
>>>> Thread(s) per core: 1
>>>> Core(s) per socket: 6
>>>> CPU socket(s): 1
>>>> Vendor ID: AuthenticAMD
>>>> CPU family: 16
>>>> Model: 8
>>>> Stepping: 1
>>>> CPU MHz: 2599.966
>>>> BogoMIPS: 5200.12
>>>> Virtualization: AMD-V
>>>>
>>>>
>>>> Thanks for your help,
>>>>
>>>> Nicola.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Tks,
>>>>> - Eric
>>>>>
>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>> two days until I start having memory problems and I need to restart
>>>>>> James.
>>>>>> The problems begin with a long series of messages such as
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>>
>>>>>> and then
>>>>>>
>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>
>>>>>> The CPU usage is over 100% and my only alternative is to restart
>>>>>> James.
>>>>>> I tried to increase VM memory (at this time I have -Xms512m
>>>>>> -Xmx1024m)
>>>>>> but the problem keeps coming back.
>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>> Thanks in advance,
>>>>>>
>>>>>>
>>>>>> Nicola.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
If you still encounter the OOM, you can further analyse with "eclipse
mat" [2] for example and ask for "suspects". It will show you the
objects that eat memory (launch james with standard Mx 512M, otherwise
you will need more time to laod dump).
Optionally, you can try to re-add the prefetchPolicy in
james-server-context.xml (see changes from [1]).
Tks,
- Eric
[1] http://s.apache.org/30x
[2] http://www.eclipse.org/mat/
On 25/05/2011 15:56, Nicola Salvemini wrote:
> Hi Eric,
> I'm using snapshot
> james-server-container-spring-3.0-M3-20110523.043027-400
>
> now I'll try to install snapshot
> james-server-container-spring-3.0-M3-20110525.115759-407
> and I'll add the complete debugging path in the startup script to
> further analyse the dump.
>
> Many thanks for your help,
>
> Nicola.
>
>
> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>> Tks for your answer.
>>
>> So you've got more or less the same configuration as I have:
>> - maildir
>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>> via smtp hook)
>> - not many users
>>
>> My config here is really stable after weeks (no oom, low memory
>> consumption when i monitor via JMX, blob-transfer folders empty).
>>
>> Yes, it sounds like some mails remains blocked, giving the issue.
>> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
>> btw, which version are you using ?
>>
>> The complete debugging path would be to add
>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>> startup script and to further analyse the dump: it will give for sure
>> the cause of oom.
>>
>> - Eric
>>
>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>
>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>> Hi,
>>> Hi,
>>>
>>>>
>>>> Can you give us a rough estimate of the traffic you have?
>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>
>>>
>>> at this time we don't have very high mail traffic, more or less 500/600
>>> mail per day.
>>> James is configured as smtp and imap servers and I think that most of
>>> the traffic is IMAP (currently there are about ten active clients at the
>>> same time making requests to the James IMAP server).
>>>
>>>> This will allow us to better define the place to look at.
>>>>
>>>> Also, is the CPU always remain 100% when you have that issue?
>>>
>>> Yes, it is in most cases.
>>>
>>>> Is the log still showing messages after the OOM?
>>>
>>> Yes, please consider following log sequence:
>>>
>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>> java.lang.OutOfMemoryError: PermGen space
>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>> noop
>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>> java.lang.OutOfMemoryError: PermGen space
>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>> java.lang.OutOfMemoryError: PermGen space
>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>> java.lang.OutOfMemoryError: PermGen space
>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>
>>>
>>> I noticed that sometimes the problem occurs for a "blockage" of
>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>> the number of mail rises too high then the problem occurs.
>>>
>>> Last thing: James is installed on a server machine whit following
>>> characteristics:
>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>> java version: 1.6.0_25
>>> RAM: 8Gb
>>> Architecture: x86_64
>>> CPU(s): 6
>>> On-line CPU(s) list: 0-5
>>> Thread(s) per core: 1
>>> Core(s) per socket: 6
>>> CPU socket(s): 1
>>> Vendor ID: AuthenticAMD
>>> CPU family: 16
>>> Model: 8
>>> Stepping: 1
>>> CPU MHz: 2599.966
>>> BogoMIPS: 5200.12
>>> Virtualization: AMD-V
>>>
>>>
>>> Thanks for your help,
>>>
>>> Nicola.
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>>>> Tks,
>>>> - Eric
>>>>
>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>> two days until I start having memory problems and I need to restart
>>>>> James.
>>>>> The problems begin with a long series of messages such as
>>>>>
>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>> | Failed to register MBean:
>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>> | Failed to register MBean:
>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>> | Failed to register MBean:
>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>> | Failed to register MBean:
>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>
>>>>> and then
>>>>>
>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>
>>>>> The CPU usage is over 100% and my only alternative is to restart James.
>>>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>>>> but the problem keeps coming back.
>>>>> Does anyone have a suggestion to solve the problem?
>>>>> Thanks in advance,
>>>>>
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
Hi Eric,
I'm using snapshot
james-server-container-spring-3.0-M3-20110523.043027-400
now I'll try to install snapshot
james-server-container-spring-3.0-M3-20110525.115759-407
and I'll add the complete debugging path in the startup script to
further analyse the dump.
Many thanks for your help,
Nicola.
On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
> Tks for your answer.
>
> So you've got more or less the same configuration as I have:
> - maildir
> - low smtp traffic (I've got a bit more with the spam, but it's rejected
> via smtp hook)
> - not many users
>
> My config here is really stable after weeks (no oom, low memory
> consumption when i monitor via JMX, blob-transfer folders empty).
>
> Yes, it sounds like some mails remains blocked, giving the issue.
> Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
> btw, which version are you using ?
>
> The complete debugging path would be to add
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
> startup script and to further analyse the dump: it will give for sure
> the cause of oom.
>
> - Eric
>
> On 25/05/2011 15:02, Nicola Salvemini wrote:
> >
> > On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> >> Hi,
> > Hi,
> >
> >>
> >> Can you give us a rough estimate of the traffic you have?
> >> Mainly SMTP? How many incoming mails/requests per second?
> >>
> >
> > at this time we don't have very high mail traffic, more or less 500/600
> > mail per day.
> > James is configured as smtp and imap servers and I think that most of
> > the traffic is IMAP (currently there are about ten active clients at the
> > same time making requests to the James IMAP server).
> >
> >> This will allow us to better define the place to look at.
> >>
> >> Also, is the CPU always remain 100% when you have that issue?
> >
> > Yes, it is in most cases.
> >
> >> Is the log still showing messages after the OOM?
> >
> > Yes, please consider following log sequence:
> >
> > DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> > java.lang.OutOfMemoryError: PermGen space
> > DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> > DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> > noop
> > DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> > org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> > DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> > java.lang.OutOfMemoryError: PermGen space
> > DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> > java.lang.OutOfMemoryError: PermGen space
> > DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> > java.lang.OutOfMemoryError: PermGen space
> > DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
> >
> >
> > I noticed that sometimes the problem occurs for a "blockage" of
> > ActiveMQ. For some reason it happens that not all spooled mail will be
> > processed and they remain in "store/activemq/blob-transfer/" folder. If
> > the number of mail rises too high then the problem occurs.
> >
> > Last thing: James is installed on a server machine whit following
> > characteristics:
> > OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> > java version: 1.6.0_25
> > RAM: 8Gb
> > Architecture: x86_64
> > CPU(s): 6
> > On-line CPU(s) list: 0-5
> > Thread(s) per core: 1
> > Core(s) per socket: 6
> > CPU socket(s): 1
> > Vendor ID: AuthenticAMD
> > CPU family: 16
> > Model: 8
> > Stepping: 1
> > CPU MHz: 2599.966
> > BogoMIPS: 5200.12
> > Virtualization: AMD-V
> >
> >
> > Thanks for your help,
> >
> > Nicola.
> >
> >
> >
> >
> >
> >
> >>
> >> Tks,
> >> - Eric
> >>
> >> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >>>
> >>> Hello,
> >>>
> >>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> >>> two days until I start having memory problems and I need to restart
> >>> James.
> >>> The problems begin with a long series of messages such as
> >>>
> >>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>> | Failed to register MBean:
> >>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>> | Failed to register MBean:
> >>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>> | Failed to register MBean:
> >>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> >>> | Failed to register MBean:
> >>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >>>
> >>> and then
> >>>
> >>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> >>> java.lang.OutOfMemoryError: PermGen space
> >>>
> >>> The CPU usage is over 100% and my only alternative is to restart James.
> >>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> >>> but the problem keeps coming back.
> >>> Does anyone have a suggestion to solve the problem?
> >>> Thanks in advance,
> >>>
> >>>
> >>> Nicola.
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >>> For additional commands, e-mail: server-user-help@james.apache.org
> >>>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> >> For additional commands, e-mail: server-user-help@james.apache.org
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Tks for your answer.
So you've got more or less the same configuration as I have:
- maildir
- low smtp traffic (I've got a bit more with the spam, but it's rejected
via smtp hook)
- not many users
My config here is really stable after weeks (no oom, low memory
consumption when i monitor via JMX, blob-transfer folders empty).
Yes, it sounds like some mails remains blocked, giving the issue.
Would it come from recent https://issues.apache.org/jira/browse/JAMES-1253 ?
btw, which version are you using ?
The complete debugging path would be to add
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
startup script and to further analyse the dump: it will give for sure
the cause of oom.
- Eric
On 25/05/2011 15:02, Nicola Salvemini wrote:
>
> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>> Hi,
> Hi,
>
>>
>> Can you give us a rough estimate of the traffic you have?
>> Mainly SMTP? How many incoming mails/requests per second?
>>
>
> at this time we don't have very high mail traffic, more or less 500/600
> mail per day.
> James is configured as smtp and imap servers and I think that most of
> the traffic is IMAP (currently there are about ten active clients at the
> same time making requests to the James IMAP server).
>
>> This will allow us to better define the place to look at.
>>
>> Also, is the CPU always remain 100% when you have that issue?
>
> Yes, it is in most cases.
>
>> Is the log still showing messages after the OOM?
>
> Yes, please consider following log sequence:
>
> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
> noop
> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>
>
> I noticed that sometimes the problem occurs for a "blockage" of
> ActiveMQ. For some reason it happens that not all spooled mail will be
> processed and they remain in "store/activemq/blob-transfer/" folder. If
> the number of mail rises too high then the problem occurs.
>
> Last thing: James is installed on a server machine whit following
> characteristics:
> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
> java version: 1.6.0_25
> RAM: 8Gb
> Architecture: x86_64
> CPU(s): 6
> On-line CPU(s) list: 0-5
> Thread(s) per core: 1
> Core(s) per socket: 6
> CPU socket(s): 1
> Vendor ID: AuthenticAMD
> CPU family: 16
> Model: 8
> Stepping: 1
> CPU MHz: 2599.966
> BogoMIPS: 5200.12
> Virtualization: AMD-V
>
>
> Thanks for your help,
>
> Nicola.
>
>
>
>
>
>
>>
>> Tks,
>> - Eric
>>
>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>
>>> Hello,
>>>
>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>> two days until I start having memory problems and I need to restart
>>> James.
>>> The problems begin with a long series of messages such as
>>>
>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>> | Failed to register MBean:
>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>> | Failed to register MBean:
>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>> | Failed to register MBean:
>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
>>> | Failed to register MBean:
>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>
>>> and then
>>>
>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> The CPU usage is over 100% and my only alternative is to restart James.
>>> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
>>> but the problem keeps coming back.
>>> Does anyone have a suggestion to solve the problem?
>>> Thanks in advance,
>>>
>>>
>>> Nicola.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Nicola Salvemini <ni...@gmail.com>.
On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
> Hi,
Hi,
>
> Can you give us a rough estimate of the traffic you have?
> Mainly SMTP? How many incoming mails/requests per second?
>
at this time we don't have very high mail traffic, more or less 500/600
mail per day.
James is configured as smtp and imap servers and I think that most of
the traffic is IMAP (currently there are about ten active clients at the
same time making requests to the James IMAP server).
> This will allow us to better define the place to look at.
>
> Also, is the CPU always remain 100% when you have that issue?
Yes, it is in most cases.
> Is the log still showing messages after the OOM?
Yes, please consider following log sequence:
DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got <tag>: 1125
DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got <command>:
noop
DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
java.lang.OutOfMemoryError: PermGen space
DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got <tag>: A05974
I noticed that sometimes the problem occurs for a "blockage" of
ActiveMQ. For some reason it happens that not all spooled mail will be
processed and they remain in "store/activemq/blob-transfer/" folder. If
the number of mail rises too high then the problem occurs.
Last thing: James is installed on a server machine whit following
characteristics:
OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
java version: 1.6.0_25
RAM: 8Gb
Architecture: x86_64
CPU(s): 6
On-line CPU(s) list: 0-5
Thread(s) per core: 1
Core(s) per socket: 6
CPU socket(s): 1
Vendor ID: AuthenticAMD
CPU family: 16
Model: 8
Stepping: 1
CPU MHz: 2599.966
BogoMIPS: 5200.12
Virtualization: AMD-V
Thanks for your help,
Nicola.
>
> Tks,
> - Eric
>
> On 25/05/2011 11:54, Nicola Salvemini wrote:
> >
> > Hello,
> >
> > I'm using James 3.0-M3 with maildir. Everything works fine for one or
> > two days until I start having memory problems and I need to restart
> > James.
> > The problems begin with a long series of messages such as
> >
> > WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> > | Failed to register MBean:
> > org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> > WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> > | Failed to register MBean:
> > org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> > WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> > | Failed to register MBean:
> > org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> > WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> > | Failed to register MBean:
> > org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> >
> > and then
> >
> > DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> > java.lang.OutOfMemoryError: PermGen space
> >
> > The CPU usage is over 100% and my only alternative is to restart James.
> > I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> > but the problem keeps coming back.
> > Does anyone have a suggestion to solve the problem?
> > Thanks in advance,
> >
> >
> > Nicola.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> > For additional commands, e-mail: server-user-help@james.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Eric Charles <er...@apache.org>.
Hi,
Can you give us a rough estimate of the traffic you have?
Mainly SMTP? How many incoming mails/requests per second?
This will allow us to better define the place to look at.
Also, is the CPU always remain 100% when you have that issue?
Is the log still showing messages after the OOM?
Tks,
- Eric
On 25/05/2011 11:54, Nicola Salvemini wrote:
>
> Hello,
>
> I'm using James 3.0-M3 with maildir. Everything works fine for one or
> two days until I start having memory problems and I need to restart
> James.
> The problems begin with a long series of messages such as
>
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
> WARN 04:46:27,928 | org.apache.activemq.broker.jmx.ManagedRegionBroker
> | Failed to register MBean:
> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>
> and then
>
> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
> java.lang.OutOfMemoryError: PermGen space
>
> The CPU usage is over 100% and my only alternative is to restart James.
> I tried to increase VM memory (at this time I have -Xms512m -Xmx1024m)
> but the problem keeps coming back.
> Does anyone have a suggestion to solve the problem?
> Thanks in advance,
>
>
> Nicola.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
> For additional commands, e-mail: server-user-help@james.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
For additional commands, e-mail: server-user-help@james.apache.org