QueueBrowser does not return large messages

[Andreas Maza <an...@gmail.com>]

Hi folks,


I am facing a problem when accessing my messages (in the DLQ) with the
queue browser: it works for "regularly-sized" messages, but it does not
return anything for large messages. I have 4 messages in my queue, where
three of them are about 20MB each. I use the ActiveMQ queue browser
implementation in my admin GUI - it shows a queue size of 4, but when I
want to browse the queue, nothing is returned and I also do not receive any
exceptions. The same behaviour applies for the ActiveMQ web console (which
runs on the same host as the ActiveMQ instance).

In general, I would prefer to load only the JMS headers when browsing a
queue for efficieny reasons. Any hints how this can be done using the queue
browse or how to avoid the above mentioned behavior?

Thanks and best regards,


Andr

Re: QueueBrowser does not return large messages

[Andreas Maza <an...@gmail.com>]

I got this solved now. It was a problem with the per-destination policy.
Adding

 <policyEntry queue="ActiveMQ.DLQ" producerFlowControl="false"
memoryLimit="500mb"/>

to the config made it work.

Best regards,
Andr

On Fri, Sep 21, 2012 at 3:37 PM, Andreas Maza <an...@gmail.com> wrote:

> Thanks for your response. I am wondering that nobody ran in such an issue
> so far. No, there is no truncation - neither in the web console nor in my
> own implementation.
>
> So your suggestions is to browse/view the messages via JMX using an
> appropriate destination policy? is this already possible now or does this
> need some code changes anyway?
>
> In JConsole, when doing a browse() all messages are returned. Some of them
> (I guess the smaller ones) contain a bodyPreview field (with a byte[255]
> since I am transferring byte[]) and some messages do not contain the
> bodyPreview field.
>
>
> Best regards,
> Andr
>
> On Tue, Sep 18, 2012 at 12:26 PM, Gary Tully <ga...@gmail.com> wrote:
>
>> I would expect some sort of truncation on the body of a large message
>> for the UI. So show only a few 100 bytes. <abc...xyz> sort of thing.
>> Like we do when we log a message.
>>
>> If you use jconsole, do you see the full 20MB?
>>
>> To implement browsing limited to headers, that would require some code
>> changes to support it. I think simplest would be a broker side
>> destination policy that limits a browse to headers only.
>>
>> There would be more work to add it as a consumer option, or maybe hack
>> it into the selector in some way.
>>
>> In any event, please raise an enhancement request, this is worth some
>> more thought and it would be easy/low impact to work in the
>> destination policy.
>>
>> On 14 September 2012 07:29, Andreas Maza <an...@gmail.com> wrote:
>> > Hi folks,
>> >
>> >
>> > I am facing a problem when accessing my messages (in the DLQ) with the
>> > queue browser: it works for "regularly-sized" messages, but it does not
>> > return anything for large messages. I have 4 messages in my queue, where
>> > three of them are about 20MB each. I use the ActiveMQ queue browser
>> > implementation in my admin GUI - it shows a queue size of 4, but when I
>> > want to browse the queue, nothing is returned and I also do not receive
>> any
>> > exceptions. The same behaviour applies for the ActiveMQ web console
>> (which
>> > runs on the same host as the ActiveMQ instance).
>> >
>> > In general, I would prefer to load only the JMS headers when browsing a
>> > queue for efficieny reasons. Any hints how this can be done using the
>> queue
>> > browse or how to avoid the above mentioned behavior?
>> >
>> > Thanks and best regards,
>> >
>> >
>> > Andr
>>
>>
>>
>> --
>> http://fusesource.com
>> http://blog.garytully.com
>>
>
>

Re: QueueBrowser does not return large messages

[Andreas Maza <an...@gmail.com>]

Thanks for your response. I am wondering that nobody ran in such an issue
so far. No, there is no truncation - neither in the web console nor in my
own implementation.

So your suggestions is to browse/view the messages via JMX using an
appropriate destination policy? is this already possible now or does this
need some code changes anyway?

In JConsole, when doing a browse() all messages are returned. Some of them
(I guess the smaller ones) contain a bodyPreview field (with a byte[255]
since I am transferring byte[]) and some messages do not contain the
bodyPreview field.


Best regards,
Andr

On Tue, Sep 18, 2012 at 12:26 PM, Gary Tully <ga...@gmail.com> wrote:

> I would expect some sort of truncation on the body of a large message
> for the UI. So show only a few 100 bytes. <abc...xyz> sort of thing.
> Like we do when we log a message.
>
> If you use jconsole, do you see the full 20MB?
>
> To implement browsing limited to headers, that would require some code
> changes to support it. I think simplest would be a broker side
> destination policy that limits a browse to headers only.
>
> There would be more work to add it as a consumer option, or maybe hack
> it into the selector in some way.
>
> In any event, please raise an enhancement request, this is worth some
> more thought and it would be easy/low impact to work in the
> destination policy.
>
> On 14 September 2012 07:29, Andreas Maza <an...@gmail.com> wrote:
> > Hi folks,
> >
> >
> > I am facing a problem when accessing my messages (in the DLQ) with the
> > queue browser: it works for "regularly-sized" messages, but it does not
> > return anything for large messages. I have 4 messages in my queue, where
> > three of them are about 20MB each. I use the ActiveMQ queue browser
> > implementation in my admin GUI - it shows a queue size of 4, but when I
> > want to browse the queue, nothing is returned and I also do not receive
> any
> > exceptions. The same behaviour applies for the ActiveMQ web console
> (which
> > runs on the same host as the ActiveMQ instance).
> >
> > In general, I would prefer to load only the JMS headers when browsing a
> > queue for efficieny reasons. Any hints how this can be done using the
> queue
> > browse or how to avoid the above mentioned behavior?
> >
> > Thanks and best regards,
> >
> >
> > Andr
>
>
>
> --
> http://fusesource.com
> http://blog.garytully.com
>

Re: QueueBrowser does not return large messages

[Gary Tully <ga...@gmail.com>]

I would expect some sort of truncation on the body of a large message
for the UI. So show only a few 100 bytes. <abc...xyz> sort of thing.
Like we do when we log a message.

If you use jconsole, do you see the full 20MB?

To implement browsing limited to headers, that would require some code
changes to support it. I think simplest would be a broker side
destination policy that limits a browse to headers only.

There would be more work to add it as a consumer option, or maybe hack
it into the selector in some way.

In any event, please raise an enhancement request, this is worth some
more thought and it would be easy/low impact to work in the
destination policy.

On 14 September 2012 07:29, Andreas Maza <an...@gmail.com> wrote:
> Hi folks,
>
>
> I am facing a problem when accessing my messages (in the DLQ) with the
> queue browser: it works for "regularly-sized" messages, but it does not
> return anything for large messages. I have 4 messages in my queue, where
> three of them are about 20MB each. I use the ActiveMQ queue browser
> implementation in my admin GUI - it shows a queue size of 4, but when I
> want to browse the queue, nothing is returned and I also do not receive any
> exceptions. The same behaviour applies for the ActiveMQ web console (which
> runs on the same host as the ActiveMQ instance).
>
> In general, I would prefer to load only the JMS headers when browsing a
> queue for efficieny reasons. Any hints how this can be done using the queue
> browse or how to avoid the above mentioned behavior?
>
> Thanks and best regards,
>
>
> Andr



-- 
http://fusesource.com
http://blog.garytully.com

Re: QueueBrowser does not return large messages

[Pawel Lacinski <la...@gmail.com>]

The problem you are facing appears beacuse of standard,  store based cursors
that works very well when  consumers are reading from the queue.  
In a few words when message comes to a queue  it is persisted in a store and
passed to a dispatch queue from which it is read by consumer. When consumer
is not fast enough or there is no consumer messages go straight to the
store. From that store they are read by cursor that buffers them. However
the number of messages that can be buffered is limited (by default is a 70%
of memory limit). 

Queue browser is actually reading only messages that are buffered by cursor.
That is why you can see only limited number of them. The rest of them is
sitting in the storage and waiting for the place in cursor. It can be
easilly tested. Add a number of messages to your queue so that not all of
them are viewed. Then from JConsole try to preview last message. It should
work. In the next step try to preview next message (its ID can be easily
calculated by a proper incrementation of a previous one). You will get null.
Then delete a message from the queue (it means you free some space in a
cursos) and try to retrieve message one more time you should get it then. 

What can be done to solve your problem is to use a different kind of cursor
/File Based Cursor/. With that cursor messages always go to both storage and
cursor. That one that are in the cursor are buffered on the disk that is why
unless you load them they don't consume memory. In order to switch on this
cursor just uncomment this in lines in policy for your queue:

/<pendingQueuePolicy>
                    <fileQueueCursor/>
</pendingQueuePolicy>/

However you have to remember that when you start to browse messages all of
them are loaded into memory. 

Detailed information about how cursor works can be found here:
http://activemq.apache.org/message-cursors.html
Here you can find destinatons policies parameters and their default values:
http://activemq.apache.org/per-destination-policies.html



--
View this message in context: http://activemq.2283324.n4.nabble.com/QueueBrowser-does-not-return-large-messages-tp4656447p4659112.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.