You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by James Strachan <ja...@gmail.com> on 2006/09/04 08:50:37 UTC
Re: Paging support
Note that MessageReferences != message contents; they are mostly just
a MessageId which is pretty small.
On 8/29/06, Fateev, Maxim <fa...@amazon.com> wrote:
> AFAIK messages are moved from journal to permantent storage (JDBC) on every checkpoint. The problem is that references for every message are still kept in memory.
> IMHO it is inherent limitation of MessageStore API as it is defined now. The only way to get message is by id. And the only way to have id is to keep it in memory.
> I'm looking into changing MessageStore API to one that instead of
>
> Message getMessage(MessageId identity)
>
> Would provide
>
> Message getNextMessage()
>
> method eliminating need to keep references in memory.
>
>
>
> -----Original Message-----
> From: Anders Bengtsson [mailto:ndrsbngtssn@yahoo.se]
> Sent: Tuesday, August 29, 2006 1:23 AM
> To: activemq-users@geronimo.apache.org
> Subject: Re: Paging support
>
>
>
> gnodet wrote:
> >
> > If you don' t use any persistent store, messages must be kept in
> > memory, so you are obvisouly limited by the available mem.
> >
>
> Yes, this is what I would expect.
>
>
>
> > If you use a jdbc store, messages can be removed from memory and store
> > for later consumption.
> >
>
> Yes, except it doesn't actually work!
>
> It writes things to the store, but it is still very much limited by memory for some reason. (Someone reported this as AMQ-845 earlier, but with no response).
>
> --
> View this message in context: http://www.nabble.com/Paging-support-tf2163517.html#a6034790
> Sent from the ActiveMQ - User forum at Nabble.com.
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: Paging support
Posted by Rob Davies <ra...@gmail.com>.
A message reference is kept in memory (which consists of a messageId
and information like sequence Id to enable faster retrieval from
storage).
There is some work currently in progress in enable paging support in
ActiveMQ - see http://issues.apache.org/activemq/browse/AMQ-845
On 5 Sep 2006, at 18:07, pradeep wrote:
>
> Maxim,
>
> Can you clarify this
> Are you storing just the message id or the whole message in memory?
> Your
> help will be highly appreciated.
>
> Thanks
> pradeep
>
>
> Fateev, Maxim wrote:
>>
>> It would be interesting to see some numbers about memory
>> utilization per
>> message in backlog.
>>
>> IMHO the main problem with keeping references in memory is that it
>> needs
>> recovery phase on broker startup. It can take a while when backlog is
>> really big.
>>
>> -----Original Message-----
>> From: James Strachan [mailto:james.strachan@gmail.com]
>> Sent: Sunday, September 03, 2006 11:51 PM
>> To: activemq-users@geronimo.apache.org
>> Subject: Re: Paging support
>>
>> Note that MessageReferences != message contents; they are mostly
>> just a
>> MessageId which is pretty small.
>>
>> On 8/29/06, Fateev, Maxim <fa...@amazon.com> wrote:
>>> AFAIK messages are moved from journal to permantent storage
>>> (JDBC) on
>>> every checkpoint. The problem is that references for every
>>> message are
>>> still kept in memory.
>>> IMHO it is inherent limitation of MessageStore API as it is
>>> defined now.
>>> The only way to get message is by id. And the only way to have id
>>> is to
>>> keep it in memory.
>>> I'm looking into changing MessageStore API to one that instead of
>>>
>>> Message getMessage(MessageId identity)
>>>
>>> Would provide
>>>
>>> Message getNextMessage()
>>>
>>> method eliminating need to keep references in memory.
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Anders Bengtsson [mailto:ndrsbngtssn@yahoo.se]
>>> Sent: Tuesday, August 29, 2006 1:23 AM
>>> To: activemq-users@geronimo.apache.org
>>> Subject: Re: Paging support
>>>
>>>
>>>
>>> gnodet wrote:
>>>>
>>>> If you don' t use any persistent store, messages must be kept in
>>>> memory, so you are obvisouly limited by the available mem.
>>>>
>>>
>>> Yes, this is what I would expect.
>>>
>>>
>>>
>>>> If you use a jdbc store, messages can be removed from memory and
>>>> store for later consumption.
>>>>
>>>
>>> Yes, except it doesn't actually work!
>>>
>>> It writes things to the store, but it is still very much limited by
>>> memory for some reason. (Someone reported this as AMQ-845
>>> earlier, but
>>> with no response).
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Paging-support-tf2163517.html#a6034790
>>> Sent from the ActiveMQ - User forum at Nabble.com.
>>>
>>
>>
>> --
>>
>> James
>> -------
>> http://radio.weblogs.com/0112098/
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Paging-support-
> tf2163517.html#a6156162
> Sent from the ActiveMQ - User forum at Nabble.com.
>
RE: Paging support
Posted by pradeep <pr...@gmail.com>.
Maxim,
Can you clarify this
Are you storing just the message id or the whole message in memory? Your
help will be highly appreciated.
Thanks
pradeep
Fateev, Maxim wrote:
>
> It would be interesting to see some numbers about memory utilization per
> message in backlog.
>
> IMHO the main problem with keeping references in memory is that it needs
> recovery phase on broker startup. It can take a while when backlog is
> really big.
>
> -----Original Message-----
> From: James Strachan [mailto:james.strachan@gmail.com]
> Sent: Sunday, September 03, 2006 11:51 PM
> To: activemq-users@geronimo.apache.org
> Subject: Re: Paging support
>
> Note that MessageReferences != message contents; they are mostly just a
> MessageId which is pretty small.
>
> On 8/29/06, Fateev, Maxim <fa...@amazon.com> wrote:
>> AFAIK messages are moved from journal to permantent storage (JDBC) on
>> every checkpoint. The problem is that references for every message are
>> still kept in memory.
>> IMHO it is inherent limitation of MessageStore API as it is defined now.
>> The only way to get message is by id. And the only way to have id is to
>> keep it in memory.
>> I'm looking into changing MessageStore API to one that instead of
>>
>> Message getMessage(MessageId identity)
>>
>> Would provide
>>
>> Message getNextMessage()
>>
>> method eliminating need to keep references in memory.
>>
>>
>>
>> -----Original Message-----
>> From: Anders Bengtsson [mailto:ndrsbngtssn@yahoo.se]
>> Sent: Tuesday, August 29, 2006 1:23 AM
>> To: activemq-users@geronimo.apache.org
>> Subject: Re: Paging support
>>
>>
>>
>> gnodet wrote:
>> >
>> > If you don' t use any persistent store, messages must be kept in
>> > memory, so you are obvisouly limited by the available mem.
>> >
>>
>> Yes, this is what I would expect.
>>
>>
>>
>> > If you use a jdbc store, messages can be removed from memory and
>> > store for later consumption.
>> >
>>
>> Yes, except it doesn't actually work!
>>
>> It writes things to the store, but it is still very much limited by
>> memory for some reason. (Someone reported this as AMQ-845 earlier, but
>> with no response).
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Paging-support-tf2163517.html#a6034790
>> Sent from the ActiveMQ - User forum at Nabble.com.
>>
>
>
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>
>
--
View this message in context: http://www.nabble.com/Paging-support-tf2163517.html#a6156162
Sent from the ActiveMQ - User forum at Nabble.com.