You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@activemq.apache.org by afei <af...@126.com> on 2009/11/09 12:24:36 UTC

Re: Out of Memory on 5.3


when a large of messages in queue,and no consumer or the consumer is very
slow, the OOM problem occur, because :
in org.apache.activemq.broker.region.Queue,the 588 line is :
 doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
,transform to :
doBrowse(false, browsedMessages, this.getMaxExpirePageSize());
 is ok.


Dejan Bosanac wrote:
> 
> Hi Mitch,
> 
> yeah, I said in thread I was referring to, that it is working with
> "regular"
> stomp connector. I started investigating AMQ-2440 patch the other day,
> should be have something soon.
> 
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
> 
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
> 
> 
> On Wed, Oct 28, 2009 at 6:18 PM, Mitch Granger
> <mi...@sophos.com>wrote:
> 
>> So we turned off stomp+nio and went back to plain old stomp and so far
>> it's
>> working fine.  New(IO) isn't always better, I guess :-)
>>
>> Seems like maybe it's this issue ->
>> https://issues.apache.org/activemq/browse/AMQ-2440
>>
>>
>> afei wrote:
>>
>>> i have same problem
>>>
>>> http://www.nabble.com/file/p26093204/aaaaaa.jpg aaaaaa.jpg
>>>
>>>
>>> themitchy wrote:
>>>
>>>> This is what we've done to tune so far:
>>>>
>>>>  - UseDedicatedTaskRunner=false
>>>>  - flow control is off
>>>>  - stomp transport uses transport.closeAsync=false
>>>>
>>>> I agree that it is because of the high number of open/close connections
>>>> from Stomp.  When we monitor through JConsole we can see more threads
>>>> starting up for each new connection.  The problem is that these threads
>>>> don't get let go.  Even though the stomp clients are disconnecting the
>>>> number of threads that get released is less than the number created. 
>>>> So
>>>>  the thread count goes up and up until it fails.  All of the above
>>>> settings/tuning only delay when it will hit the wall.
>>>>
>>>> Dejan Bosanac wrote:
>>>>
>>>>> Hi Mitch,
>>>>>
>>>>> I think the root cause of this problem is that you probably have Stomp
>>>>> clients that open/close connection at a high rate. I simulated this
>>>>> problem
>>>>> on OSX with a StompLoadTest (
>>>>>
>>>>> http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/transport/stomp/StompLoadTest.java?view=log
>>>>> ),
>>>>> while trying to reproduce "too many open files" problem. You can find
>>>>> some
>>>>> of my findings (and workaround) in this thread.
>>>>>
>>>>>
>>>>> http://www.nabble.com/%22too-many-open-files%22-error-with-5.3-and-Stomp-tt25888831.html#a26010080
>>>>>
>>>>> (BTW. it is producing "too many open files" problem on linux)
>>>>> Basically,
>>>>> the
>>>>> problem with stomp is that every send is done in separate connection
>>>>> and
>>>>> thus considered to be a new producer for every message. So when
>>>>> producer
>>>>> flow control is hit, the producers are piling up and probably not
>>>>> releasing
>>>>> connections. Thus you can observe large number of tcp connections on
>>>>> the
>>>>> system in state TIME_WAIT (and TIME_CLOSE), which causes that the
>>>>> system
>>>>> limit is hit at one point. In the above thread, you can find a
>>>>> workaround
>>>>> that worked for me for that test. I started investigating this more
>>>>> and
>>>>> hopefully I'll have some more findings in the near future.
>>>>>
>>>>> Cheers
>>>>> --
>>>>> Dejan Bosanac - http://twitter.com/dejanb
>>>>>
>>>>> Open Source Integration - http://fusesource.com/
>>>>> ActiveMQ in Action - http://www.manning.com/snyder/
>>>>> Blog - http://www.nighttale.net
>>>>>
>>>>>
>>>>> On Tue, Oct 27, 2009 at 1:07 AM, Mitch Granger
>>>>> <mi...@sophos.com>wrote:
>>>>>
>>>>>  Update: We've [nearly] proven that this only happens with AMQ running
>>>>>> on
>>>>>> openVZ.  What exactly is causing it, we're still not sure.  After
>>>>>> memoryUsage is met, the number of threads skyrockets until we get
>>>>>> OutOfMemoryError.
>>>>>>
>>>>>> It works just fine on regular hardware; We're going to try VMWare
>>>>>> tomorrow.
>>>>>>
>>>>>> One thing really worth mentioning is that by using the fileCursor we
>>>>>> actually started seeing it use the Temp Store.  When reading about
>>>>>> systemUsage it is NOT intuitive that the Temp Store does not come
>>>>>> into
>>>>>> play
>>>>>> with the default cursor.  Anyone keeping a significant volume of
>>>>>> messages on
>>>>>> their queues should be well served by changing the cursor.
>>>>>>
>>>>>>
>>>>>> Mitch Granger wrote:
>>>>>>
>>>>>>  Config is attached.  We have also tried the activemq-scalability.xml
>>>>>>> with
>>>>>>> the only change being adding a stomp connector.
>>>>>>>
>>>>>>> Once we hit the memoryUsage limit we can [sometimes] connect new
>>>>>>> consumers
>>>>>>> but nothing comes back after we send the SUBSCRIBE frame.
>>>>>>>
>>>>>>> I expect sending to fail when we hit this limit but if we can't
>>>>>>> subscribe
>>>>>>> there's no chance of recovering from this state.
>>>>>>>
>>>>>>> Rob Davies wrote:
>>>>>>>
>>>>>>>  On 26 Oct 2009, at 17:38, themitchy wrote:
>>>>>>>>
>>>>>>>>  We're using only persistent messages and heap size is set to 2GB
>>>>>>>> yet
>>>>>>>> we
>>>>>>>>
>>>>>>>>> hit
>>>>>>>>> the memoryUsage limit quite quickly (system usage config below).
>>>>>>>>> This
>>>>>>>>> is
>>>>>>>>> followed by "java.lang.OutOfMemoryError: unable to create new
>>>>>>>>> native
>>>>>>>>>  thread"
>>>>>>>>> as the process quickly reaches the 2GB of heap we gave it.  How
>>>>>>>>> are
>>>>>>>>> we
>>>>>>>>> getting to that point with the memoryUsage limit set far below it?
>>>>>>>>>
>>>>>>>>> Is there no way to get AMQ to gracefully limit it's memory usage?
>>>>>>>>>
>>>>>>>>>      <systemUsage>
>>>>>>>>>          <systemUsage>
>>>>>>>>>              <memoryUsage>
>>>>>>>>>                  <memoryUsage limit="256 mb"/>
>>>>>>>>>              </memoryUsage>
>>>>>>>>>              <storeUsage>
>>>>>>>>>                  <storeUsage limit="60 gb" name="foo"/>
>>>>>>>>>              </storeUsage>
>>>>>>>>>              <tempUsage>
>>>>>>>>>                  <tempUsage limit="60 gb"/>
>>>>>>>>>              </tempUsage>
>>>>>>>>>          </systemUsage>
>>>>>>>>>      </systemUsage>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://www.nabble.com/Out-of-Memory-on-5.3-tp26064098p26064098.html
>>>>>>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>  Can you send the rest of your config ?
>>>>>>>>
>>>>>>>> Rob Davies
>>>>>>>> http://twitter.com/rajdavies
>>>>>>>> I work here: http://fusesource.com
>>>>>>>> My Blog: http://rajdavies.blogspot.com/
>>>>>>>> I'm writing this: http://www.manning.com/snyder/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>
>>>
> 
> 
> -----
> Dejan Bosanac
> 
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
> 
http://old.nabble.com/file/p26264779/dump.jpg 
-- 
View this message in context: http://old.nabble.com/Out-of-Memory-on-5.3-tp26064098p26264779.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.