You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by Eric Charles <er...@apache.org> on 2011/05/25 16:18:11 UTC
Re: ActiveMQ and memory problems
Sorry, I was meaning "remove the prefetch policy".
For mails blocked in queues, you can read more on
http://markmail.org/message/v4zxfkbkjjllwrbb and try get more
information from the jmx console for those messages and on the activemq
mbeans.
Tks,
- Eric
On 25/05/2011 16:09, Eric Charles wrote:
> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>
> If you still encounter the OOM, you can further analyse with "eclipse
> mat" [2] for example and ask for "suspects". It will show you the
> objects that eat memory (launch james with standard Mx 512M, otherwise
> you will need more time to laod dump).
>
> Optionally, you can try to re-add the prefetchPolicy in
> james-server-context.xml (see changes from [1]).
>
> Tks,
> - Eric
>
> [1] http://s.apache.org/30x
> [2] http://www.eclipse.org/mat/
>
> On 25/05/2011 15:56, Nicola Salvemini wrote:
>> Hi Eric,
>> I'm using snapshot
>> james-server-container-spring-3.0-M3-20110523.043027-400
>>
>> now I'll try to install snapshot
>> james-server-container-spring-3.0-M3-20110525.115759-407
>> and I'll add the complete debugging path in the startup script to
>> further analyse the dump.
>>
>> Many thanks for your help,
>>
>> Nicola.
>>
>>
>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>> Tks for your answer.
>>>
>>> So you've got more or less the same configuration as I have:
>>> - maildir
>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>> via smtp hook)
>>> - not many users
>>>
>>> My config here is really stable after weeks (no oom, low memory
>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>
>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>> Would it come from recent
>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>> btw, which version are you using ?
>>>
>>> The complete debugging path would be to add
>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>> startup script and to further analyse the dump: it will give for sure
>>> the cause of oom.
>>>
>>> - Eric
>>>
>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>
>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>> Hi,
>>>> Hi,
>>>>
>>>>>
>>>>> Can you give us a rough estimate of the traffic you have?
>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>
>>>>
>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>> mail per day.
>>>> James is configured as smtp and imap servers and I think that most of
>>>> the traffic is IMAP (currently there are about ten active clients at
>>>> the
>>>> same time making requests to the James IMAP server).
>>>>
>>>>> This will allow us to better define the place to look at.
>>>>>
>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>
>>>> Yes, it is in most cases.
>>>>
>>>>> Is the log still showing messages after the OOM?
>>>>
>>>> Yes, please consider following log sequence:
>>>>
>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>> noop
>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>> java.lang.OutOfMemoryError: PermGen space
>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>
>>>>
>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>> the number of mail rises too high then the problem occurs.
>>>>
>>>> Last thing: James is installed on a server machine whit following
>>>> characteristics:
>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>> java version: 1.6.0_25
>>>> RAM: 8Gb
>>>> Architecture: x86_64
>>>> CPU(s): 6
>>>> On-line CPU(s) list: 0-5
>>>> Thread(s) per core: 1
>>>> Core(s) per socket: 6
>>>> CPU socket(s): 1
>>>> Vendor ID: AuthenticAMD
>>>> CPU family: 16
>>>> Model: 8
>>>> Stepping: 1
>>>> CPU MHz: 2599.966
>>>> BogoMIPS: 5200.12
>>>> Virtualization: AMD-V
>>>>
>>>>
>>>> Thanks for your help,
>>>>
>>>> Nicola.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>>>> Tks,
>>>>> - Eric
>>>>>
>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>> two days until I start having memory problems and I need to restart
>>>>>> James.
>>>>>> The problems begin with a long series of messages such as
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>> WARN 04:46:27,928 |
>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>> | Failed to register MBean:
>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>
>>>>>>
>>>>>> and then
>>>>>>
>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>
>>>>>> The CPU usage is over 100% and my only alternative is to restart
>>>>>> James.
>>>>>> I tried to increase VM memory (at this time I have -Xms512m
>>>>>> -Xmx1024m)
>>>>>> but the problem keeps coming back.
>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>> Thanks in advance,
>>>>>>
>>>>>>
>>>>>> Nicola.
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>> For additional commands, e-mail: server-user-help@james.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org
Re: ActiveMQ and memory problems
Posted by Norman Maurer <no...@googlemail.com>.
Removing the prefetch config will not do anything good...
Bye
Norman
2011/5/25, Eric Charles <er...@apache.org>:
> Sorry, I was meaning "remove the prefetch policy".
>
> For mails blocked in queues, you can read more on
> http://markmail.org/message/v4zxfkbkjjllwrbb and try get more
> information from the jmx console for those messages and on the activemq
> mbeans.
>
> Tks,
> - Eric
>
> On 25/05/2011 16:09, Eric Charles wrote:
>> OK, so you're using a snapshot of 23/5, just after the commit [1] of 20/5.
>>
>> If you still encounter the OOM, you can further analyse with "eclipse
>> mat" [2] for example and ask for "suspects". It will show you the
>> objects that eat memory (launch james with standard Mx 512M, otherwise
>> you will need more time to laod dump).
>>
>> Optionally, you can try to re-add the prefetchPolicy in
>> james-server-context.xml (see changes from [1]).
>>
>> Tks,
>> - Eric
>>
>> [1] http://s.apache.org/30x
>> [2] http://www.eclipse.org/mat/
>>
>> On 25/05/2011 15:56, Nicola Salvemini wrote:
>>> Hi Eric,
>>> I'm using snapshot
>>> james-server-container-spring-3.0-M3-20110523.043027-400
>>>
>>> now I'll try to install snapshot
>>> james-server-container-spring-3.0-M3-20110525.115759-407
>>> and I'll add the complete debugging path in the startup script to
>>> further analyse the dump.
>>>
>>> Many thanks for your help,
>>>
>>> Nicola.
>>>
>>>
>>> On Wed, 2011-05-25 at 15:17 +0200, Eric Charles wrote:
>>>> Tks for your answer.
>>>>
>>>> So you've got more or less the same configuration as I have:
>>>> - maildir
>>>> - low smtp traffic (I've got a bit more with the spam, but it's rejected
>>>> via smtp hook)
>>>> - not many users
>>>>
>>>> My config here is really stable after weeks (no oom, low memory
>>>> consumption when i monitor via JMX, blob-transfer folders empty).
>>>>
>>>> Yes, it sounds like some mails remains blocked, giving the issue.
>>>> Would it come from recent
>>>> https://issues.apache.org/jira/browse/JAMES-1253 ?
>>>> btw, which version are you using ?
>>>>
>>>> The complete debugging path would be to add
>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your_path in the
>>>> startup script and to further analyse the dump: it will give for sure
>>>> the cause of oom.
>>>>
>>>> - Eric
>>>>
>>>> On 25/05/2011 15:02, Nicola Salvemini wrote:
>>>>>
>>>>> On Wed, 2011-05-25 at 12:16 +0200, Eric Charles wrote:
>>>>>> Hi,
>>>>> Hi,
>>>>>
>>>>>>
>>>>>> Can you give us a rough estimate of the traffic you have?
>>>>>> Mainly SMTP? How many incoming mails/requests per second?
>>>>>>
>>>>>
>>>>> at this time we don't have very high mail traffic, more or less 500/600
>>>>> mail per day.
>>>>> James is configured as smtp and imap servers and I think that most of
>>>>> the traffic is IMAP (currently there are about ten active clients at
>>>>> the
>>>>> same time making requests to the James IMAP server).
>>>>>
>>>>>> This will allow us to better define the place to look at.
>>>>>>
>>>>>> Also, is the CPU always remain 100% when you have that issue?
>>>>>
>>>>> Yes, it is in most cases.
>>>>>
>>>>>> Is the log still showing messages after the OOM?
>>>>>
>>>>> Yes, please consider following log sequence:
>>>>>
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<tag>: 1125
>>>>> DEBUG 04:51:16,731 | james.imapserver | ID=2085331846 Got<command>:
>>>>> noop
>>>>> DEBUG 04:51:16,731 | james.smtpserver | ID=1200023601 executing hook
>>>>> org.apache.james.protocols.smtp.core.log.HookResultLogger@5ae99cbb
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:16,731 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.smtpserver | Unable to process request
>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>> DEBUG 04:51:13,425 | james.imapserver | ID=2140019352 Got<tag>: A05974
>>>>>
>>>>>
>>>>> I noticed that sometimes the problem occurs for a "blockage" of
>>>>> ActiveMQ. For some reason it happens that not all spooled mail will be
>>>>> processed and they remain in "store/activemq/blob-transfer/" folder. If
>>>>> the number of mail rises too high then the problem occurs.
>>>>>
>>>>> Last thing: James is installed on a server machine whit following
>>>>> characteristics:
>>>>> OS: Opensuse 11.4 64bit Linux kernel 2.6.37.6-0.5
>>>>> java version: 1.6.0_25
>>>>> RAM: 8Gb
>>>>> Architecture: x86_64
>>>>> CPU(s): 6
>>>>> On-line CPU(s) list: 0-5
>>>>> Thread(s) per core: 1
>>>>> Core(s) per socket: 6
>>>>> CPU socket(s): 1
>>>>> Vendor ID: AuthenticAMD
>>>>> CPU family: 16
>>>>> Model: 8
>>>>> Stepping: 1
>>>>> CPU MHz: 2599.966
>>>>> BogoMIPS: 5200.12
>>>>> Virtualization: AMD-V
>>>>>
>>>>>
>>>>> Thanks for your help,
>>>>>
>>>>> Nicola.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Tks,
>>>>>> - Eric
>>>>>>
>>>>>> On 25/05/2011 11:54, Nicola Salvemini wrote:
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I'm using James 3.0-M3 with maildir. Everything works fine for one or
>>>>>>> two days until I start having memory problems and I need to restart
>>>>>>> James.
>>>>>>> The problems begin with a long series of messages such as
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>> WARN 04:46:27,928 |
>>>>>>> org.apache.activemq.broker.jmx.ManagedRegionBroker
>>>>>>> | Failed to register MBean:
>>>>>>> org.apache.activemq:BrokerName=james,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=spool,clientId=ID_alice-58445-1306246275420-4_1,consumerId=ID_alice-58445-1306246275420-3_1_22_4524
>>>>>>>
>>>>>>>
>>>>>>> and then
>>>>>>>
>>>>>>> DEBUG 04:51:11,445 | james.smtpserver | Unable to process request
>>>>>>> java.lang.OutOfMemoryError: PermGen space
>>>>>>>
>>>>>>> The CPU usage is over 100% and my only alternative is to restart
>>>>>>> James.
>>>>>>> I tried to increase VM memory (at this time I have -Xms512m
>>>>>>> -Xmx1024m)
>>>>>>> but the problem keeps coming back.
>>>>>>> Does anyone have a suggestion to solve the problem?
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>>
>>>>>>> Nicola.
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: server-user-unsubscribe@james.apache.org
>>> For additional commands, e-mail: server-user-help@james.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
> For additional commands, e-mail: server-dev-help@james.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org