You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by jahlborn <ja...@gmail.com> on 2016/12/15 16:09:12 UTC
Custom code to browse messages in the message store?
We have activemq embedded in our product. Since it runs in the same jvm, we
have various resource limits in place. These limits make the browsing
functionality somewhat less than ideal (you can only read so many messages
in a large queue). I was wondering if someone who has deeper knowledge of
activemq could suggest a way to read messages directly from the data store
(or some other way to do unbounded browsing which doesn't use up all system
memory).
I already attempted this a bit myself by adding a method to the region Queue
for first pulling from the "paged in" messages, and then directly from the
"PendingMessageCursor messages" to read messages. However, despite the fact
that the cursor is backed by the message store, it still seems to want to
"page in" messages, and also abides by the resource limits. (i was
borrowing code from Queue.getMessage(String id)).
Is there any way to "bypass" the "paging in" logic and just read messages
directly from the message store? i'm okay with this being "slow", i'd
prefer to have a solution which works slowly rather than no solution at
all...
--
View this message in context: http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Custom code to browse messages in the message store?
Posted by jahlborn <ja...@gmail.com>.
oh wow! that worked beautifully! at first i was trying to deal with
messages that had already been paged in, but then i realized that anything
which has not yet been consumed should still be in the store. so it really
was as simple as passing a MessageRecoveryListener to the store and I can
browse the entire message list.
thanks for your help!
christopher.l.shannon wrote
> The transaction you are referring to is a transaction on the KahaDB
> index and a new one will be created by KahaDB when calling that method
> so you don't need to worry about it. In terms of statefulness, you
> are correct that the order index tracks state of where it left off.
> But I believe it is safe to use the recover() method (which always
> resets the cursor and starts at the beginning) as there are things
> like duplicate detection in place so it should be able to handle it.
>
> In fact, if you call browse() on a Topic from JMX it will eventually
> end up delegating to the recover method on the Topic Store and it will
> return up to the number of messages of the browse page size so it has
> been designed to be called multiple times because it will be called
> every time you call browse(). Here is how it is used and might be a
> good example for basing your code off of:
> https://github.com/apache/activemq/blob/activemq-5.14.2/activemq-broker/src/main/java/org/apache/activemq/broker/region/Topic.java#L638
> For recoverNextMessages I don't remember if it would work or not
> without taking a closer look.
>
> Unfortunately I don't think there is really any other methods right
> now to try and iterate over the messages in the store. The best thing
> to do would be to write a quick test and see if it works for you.
>
> On Fri, Dec 16, 2016 at 3:26 PM, jahlborn <
> jahlborn@
> > wrote:
>> So, digging into the message store recoverNextMessages() code, it's tough
>> to
>> tell what is going in in terms of statefulness. That method certainly
>> seems
>> like it would get me the information that i'm after. two concerns: 1. it
>> looks to be expecting an existing Transaction (maybe i already have one
>> in
>> place since i'm working off of an existing session). 2.
>> "recoverNextMessages" seems to imply that it's basing the notion of
>> "next"
>> off of current state and calling this method will change that state. do
>> i
>> need to do something with the store state to ensure that i don't mess up
>> this state for normal message consumption?
>>
>> thanks for your help so far!
>>
>>
>> christopher.l.shannon wrote
>>> You should be able to implement your own MessageRecoveryListener and
>>> use the recover() or recoverNextMessages() methods on the MessageStore
>>> interface. The MessageStore object is part of the region Queue (it is
>>> inherited from the parent BaseDestination class)
>>> http://activemq.apache.org/maven/apidocs/org/apache/activemq/store/MessageStore.html
>>>
>>> Calling that method will iterate over each message in the store, load
>>> it into memory and then and pass it to the MessageRecoveryListener
>>> that you provide. If you are using KahaDB then calling these methods
>>> will lock the index and prevent other writes/reads from the store
>>> while executing the method.
>>>
>>> On Thu, Dec 15, 2016 at 11:09 AM, jahlborn <
>>
>>> jahlborn@
>>
>>> > wrote:
>>>> We have activemq embedded in our product. Since it runs in the same
>>>> jvm,
>>>> we
>>>> have various resource limits in place. These limits make the browsing
>>>> functionality somewhat less than ideal (you can only read so many
>>>> messages
>>>> in a large queue). I was wondering if someone who has deeper knowledge
>>>> of
>>>> activemq could suggest a way to read messages directly from the data
>>>> store
>>>> (or some other way to do unbounded browsing which doesn't use up all
>>>> system
>>>> memory).
>>>>
>>>> I already attempted this a bit myself by adding a method to the region
>>>> Queue
>>>> for first pulling from the "paged in" messages, and then directly from
>>>> the
>>>> "PendingMessageCursor messages" to read messages. However, despite the
>>>> fact
>>>> that the cursor is backed by the message store, it still seems to want
>>>> to
>>>> "page in" messages, and also abides by the resource limits. (i was
>>>> borrowing code from Queue.getMessage(String id)).
>>>>
>>>> Is there any way to "bypass" the "paging in" logic and just read
>>>> messages
>>>> directly from the message store? i'm okay with this being "slow", i'd
>>>> prefer to have a solution which works slowly rather than no solution at
>>>> all...
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423p4720542.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423p4720593.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Custom code to browse messages in the message store?
Posted by Christopher Shannon <ch...@gmail.com>.
The transaction you are referring to is a transaction on the KahaDB
index and a new one will be created by KahaDB when calling that method
so you don't need to worry about it. In terms of statefulness, you
are correct that the order index tracks state of where it left off.
But I believe it is safe to use the recover() method (which always
resets the cursor and starts at the beginning) as there are things
like duplicate detection in place so it should be able to handle it.
In fact, if you call browse() on a Topic from JMX it will eventually
end up delegating to the recover method on the Topic Store and it will
return up to the number of messages of the browse page size so it has
been designed to be called multiple times because it will be called
every time you call browse(). Here is how it is used and might be a
good example for basing your code off of:
https://github.com/apache/activemq/blob/activemq-5.14.2/activemq-broker/src/main/java/org/apache/activemq/broker/region/Topic.java#L638
For recoverNextMessages I don't remember if it would work or not
without taking a closer look.
Unfortunately I don't think there is really any other methods right
now to try and iterate over the messages in the store. The best thing
to do would be to write a quick test and see if it works for you.
On Fri, Dec 16, 2016 at 3:26 PM, jahlborn <ja...@gmail.com> wrote:
> So, digging into the message store recoverNextMessages() code, it's tough to
> tell what is going in in terms of statefulness. That method certainly seems
> like it would get me the information that i'm after. two concerns: 1. it
> looks to be expecting an existing Transaction (maybe i already have one in
> place since i'm working off of an existing session). 2.
> "recoverNextMessages" seems to imply that it's basing the notion of "next"
> off of current state and calling this method will change that state. do i
> need to do something with the store state to ensure that i don't mess up
> this state for normal message consumption?
>
> thanks for your help so far!
>
>
> christopher.l.shannon wrote
>> You should be able to implement your own MessageRecoveryListener and
>> use the recover() or recoverNextMessages() methods on the MessageStore
>> interface. The MessageStore object is part of the region Queue (it is
>> inherited from the parent BaseDestination class)
>> http://activemq.apache.org/maven/apidocs/org/apache/activemq/store/MessageStore.html
>>
>> Calling that method will iterate over each message in the store, load
>> it into memory and then and pass it to the MessageRecoveryListener
>> that you provide. If you are using KahaDB then calling these methods
>> will lock the index and prevent other writes/reads from the store
>> while executing the method.
>>
>> On Thu, Dec 15, 2016 at 11:09 AM, jahlborn <
>
>> jahlborn@
>
>> > wrote:
>>> We have activemq embedded in our product. Since it runs in the same jvm,
>>> we
>>> have various resource limits in place. These limits make the browsing
>>> functionality somewhat less than ideal (you can only read so many
>>> messages
>>> in a large queue). I was wondering if someone who has deeper knowledge
>>> of
>>> activemq could suggest a way to read messages directly from the data
>>> store
>>> (or some other way to do unbounded browsing which doesn't use up all
>>> system
>>> memory).
>>>
>>> I already attempted this a bit myself by adding a method to the region
>>> Queue
>>> for first pulling from the "paged in" messages, and then directly from
>>> the
>>> "PendingMessageCursor messages" to read messages. However, despite the
>>> fact
>>> that the cursor is backed by the message store, it still seems to want to
>>> "page in" messages, and also abides by the resource limits. (i was
>>> borrowing code from Queue.getMessage(String id)).
>>>
>>> Is there any way to "bypass" the "paging in" logic and just read messages
>>> directly from the message store? i'm okay with this being "slow", i'd
>>> prefer to have a solution which works slowly rather than no solution at
>>> all...
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423p4720542.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Custom code to browse messages in the message store?
Posted by jahlborn <ja...@gmail.com>.
So, digging into the message store recoverNextMessages() code, it's tough to
tell what is going in in terms of statefulness. That method certainly seems
like it would get me the information that i'm after. two concerns: 1. it
looks to be expecting an existing Transaction (maybe i already have one in
place since i'm working off of an existing session). 2.
"recoverNextMessages" seems to imply that it's basing the notion of "next"
off of current state and calling this method will change that state. do i
need to do something with the store state to ensure that i don't mess up
this state for normal message consumption?
thanks for your help so far!
christopher.l.shannon wrote
> You should be able to implement your own MessageRecoveryListener and
> use the recover() or recoverNextMessages() methods on the MessageStore
> interface. The MessageStore object is part of the region Queue (it is
> inherited from the parent BaseDestination class)
> http://activemq.apache.org/maven/apidocs/org/apache/activemq/store/MessageStore.html
>
> Calling that method will iterate over each message in the store, load
> it into memory and then and pass it to the MessageRecoveryListener
> that you provide. If you are using KahaDB then calling these methods
> will lock the index and prevent other writes/reads from the store
> while executing the method.
>
> On Thu, Dec 15, 2016 at 11:09 AM, jahlborn <
> jahlborn@
> > wrote:
>> We have activemq embedded in our product. Since it runs in the same jvm,
>> we
>> have various resource limits in place. These limits make the browsing
>> functionality somewhat less than ideal (you can only read so many
>> messages
>> in a large queue). I was wondering if someone who has deeper knowledge
>> of
>> activemq could suggest a way to read messages directly from the data
>> store
>> (or some other way to do unbounded browsing which doesn't use up all
>> system
>> memory).
>>
>> I already attempted this a bit myself by adding a method to the region
>> Queue
>> for first pulling from the "paged in" messages, and then directly from
>> the
>> "PendingMessageCursor messages" to read messages. However, despite the
>> fact
>> that the cursor is backed by the message store, it still seems to want to
>> "page in" messages, and also abides by the resource limits. (i was
>> borrowing code from Queue.getMessage(String id)).
>>
>> Is there any way to "bypass" the "paging in" logic and just read messages
>> directly from the message store? i'm okay with this being "slow", i'd
>> prefer to have a solution which works slowly rather than no solution at
>> all...
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
View this message in context: http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423p4720542.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Custom code to browse messages in the message store?
Posted by Christopher Shannon <ch...@gmail.com>.
You should be able to implement your own MessageRecoveryListener and
use the recover() or recoverNextMessages() methods on the MessageStore
interface. The MessageStore object is part of the region Queue (it is
inherited from the parent BaseDestination class)
http://activemq.apache.org/maven/apidocs/org/apache/activemq/store/MessageStore.html
Calling that method will iterate over each message in the store, load
it into memory and then and pass it to the MessageRecoveryListener
that you provide. If you are using KahaDB then calling these methods
will lock the index and prevent other writes/reads from the store
while executing the method.
On Thu, Dec 15, 2016 at 11:09 AM, jahlborn <ja...@gmail.com> wrote:
> We have activemq embedded in our product. Since it runs in the same jvm, we
> have various resource limits in place. These limits make the browsing
> functionality somewhat less than ideal (you can only read so many messages
> in a large queue). I was wondering if someone who has deeper knowledge of
> activemq could suggest a way to read messages directly from the data store
> (or some other way to do unbounded browsing which doesn't use up all system
> memory).
>
> I already attempted this a bit myself by adding a method to the region Queue
> for first pulling from the "paged in" messages, and then directly from the
> "PendingMessageCursor messages" to read messages. However, despite the fact
> that the cursor is backed by the message store, it still seems to want to
> "page in" messages, and also abides by the resource limits. (i was
> borrowing code from Queue.getMessage(String id)).
>
> Is there any way to "bypass" the "paging in" logic and just read messages
> directly from the message store? i'm okay with this being "slow", i'd
> prefer to have a solution which works slowly rather than no solution at
> all...
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Custom-code-to-browse-messages-in-the-message-store-tp4720423.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.