You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Christoph Kutzinski <ku...@gmx.de> on 2010/01/19 10:44:27 UTC

Problem with 'dispatched queue'

Hi,

let me first describe my use-case as it is probably a little bit different from the usual JMS use case:

We have a single process which fills a JMS queue from a pool of available items (say we have 50 items total).
The we have n consumers (lets say 2) which read the items from the queue in a synchronous way (i.e. Spring JmsTemplate#receive).
This goes like so:
- consumer A fetches 5 items
- consumer B fetches 20 items
- consumer A fetches 5 items
and so on.

The problem we have now is that we are not able to fetch all items from the queue - e.g. if there are still 10 items left in the queue and I want to get 10, I only get 5 or so.

I've already found the prefetchSize configuration and set it to 0 (also tried 1), but still the same problem.
The problem seems to be that each consumer has an internal 'dispatched queue' which already has some entries and consumer A has no possibility to access the dispatched queue of consumer B.

For example the web console looks like this:
Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5; Prefetch 0; Max Pending 0

Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue 10; Prefetch 0; Max Pending 0

In this case there would be e.g. still 15 items total in the queue, but with consumer A I would onyl be able to get 5 items and with consumer B only 10 items.

I've tried various configuration options which I found in the ActiveMQ docu (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none seemed to help.

Can anyone help? Does anyone know how to configure/disable this dispatched queue?


thanks
Christoph

Re: Problem with 'dispatched queue'

Posted by Gary Tully <ga...@gmail.com>.
maybe take the time to write a simple test case to isolate the problem

2010/1/21 Christoph Kutzinski <ku...@gmx.de>

>
> http://old.nabble.com/Problem-with-%27dispatched-queue%27-td27223270.html
>
> No one got an idea what's wrong here?
> Any directions where I could get more information? E.g.
> - logging output
> - JMX console
> - source code?
>
>
> Christoph
>
>
> --
> View this message in context:
> http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27255226.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: Problem with 'dispatched queue'

Posted by Christoph Kutzinski <ku...@gmx.de>.
http://old.nabble.com/Problem-with-%27dispatched-queue%27-td27223270.html

No one got an idea what's wrong here?
Any directions where I could get more information? E.g.
- logging output
- JMX console
- source code?


Christoph


-- 
View this message in context: http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27255226.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with 'dispatched queue'

Posted by Christoph Kutzinski <ku...@gmx.de>.
An additional observation:

The size of the dispacthed queue seems also to be equal the (largest) number
of messages I received in a row. E.g. if I received 5 message the webconsole
looks like this:

"Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5"

If I receive 10 messages later from the same consumer:

"Enqueues 25; Dequeues 15; Dispatched 25; Dispatched Queue 10"


Also, there always seem to be exactly 2 consumers (for one receiving) VM.
Don't know if that's important, I guess that's a Spring
SingleConnectionFactory 'magic'
-- 
View this message in context: http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27226129.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with 'dispatched queue'

Posted by Christoph Kutzinski <ku...@gmx.de>.
No,

sorry no simple testcase and currently don't have the time to create one.

As I can the see from the 'Prefetch 0' in the web console, prefetchSize
seems to get noticed from ActiveMQ. However, it doesn't seem to have any
effect on the dispatched queue.


I can only provide this code snippet currently on how I use the queue:


    private String getDestinationName(Pool pool) {
        return QUEUE_PREFIX + pool.getUniqueName() +
"?consumer.prefetchSize=0";
    }
    
    private List<Item> reserveItem(Pool pool, int number) throws
ServiceException {
        List<Pool> reserved = Lists.newArrayListWithExpectedSize(number);

        this.JmsTemplate.setReceiveTimeout(2000);

        String queueName = getDestinationName(pool);
        
        for (int i = 0; i < number; i++) {
            TextMessage message = (TextMessage)
this.jmsTemplate.receive(queueName);
            
            if (message == null) {
                throw ...  <== that's what happens when no messages cannot
be fetched anymore from queue
            }
            
            try {
                String idStr = message.getText();
                long itemId = Long.parseLong(idStr);

                Item item = reserveItem(itemId);
                reserved.add(item);
                
            } catch (Exception e) {
                throw ...
            }
        }
        
        return reserved;
    }

thanks
Christoph



Gary Tully wrote:
> 
> prefetch==0 should be the answer here. Do you have a simple test case you
> can share?
> 
> 2010/1/19 Christoph Kutzinski <ku...@gmx.de>
> 
>> Hi,
>>
>> let me first describe my use-case as it is probably a little bit
>> different
>> from the usual JMS use case:
>>
>> We have a single process which fills a JMS queue from a pool of available
>> items (say we have 50 items total).
>> The we have n consumers (lets say 2) which read the items from the queue
>> in
>> a synchronous way (i.e. Spring JmsTemplate#receive).
>> This goes like so:
>> - consumer A fetches 5 items
>> - consumer B fetches 20 items
>> - consumer A fetches 5 items
>> and so on.
>>
>> The problem we have now is that we are not able to fetch all items from
>> the
>> queue - e.g. if there are still 10 items left in the queue and I want to
>> get
>> 10, I only get 5 or so.
>>
>> I've already found the prefetchSize configuration and set it to 0 (also
>> tried 1), but still the same problem.
>> The problem seems to be that each consumer has an internal 'dispatched
>> queue' which already has some entries and consumer A has no possibility
>> to
>> access the dispatched queue of consumer B.
>>
>> For example the web console looks like this:
>> Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5;
>> Prefetch 0; Max Pending 0
>>
>> Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue 10;
>> Prefetch 0; Max Pending 0
>>
>> In this case there would be e.g. still 15 items total in the queue, but
>> with consumer A I would onyl be able to get 5 items and with consumer B
>> only
>> 10 items.
>>
>> I've tried various configuration options which I found in the ActiveMQ
>> docu
>> (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none seemed to
>> help.
>>
>> Can anyone help? Does anyone know how to configure/disable this
>> dispatched
>> queue?
>>
>>
>> thanks
>> Christoph
>>
> 
> 
> 
> -- 
> http://blog.garytully.com
> 
> Open Source Integration
> http://fusesource.com
> 
> 

-- 
View this message in context: http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27225878.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with 'dispatched queue'

Posted by Gary Tully <ga...@gmail.com>.
prefetch==0 should be the answer here. Do you have a simple test case you
can share?

2010/1/19 Christoph Kutzinski <ku...@gmx.de>

> Hi,
>
> let me first describe my use-case as it is probably a little bit different
> from the usual JMS use case:
>
> We have a single process which fills a JMS queue from a pool of available
> items (say we have 50 items total).
> The we have n consumers (lets say 2) which read the items from the queue in
> a synchronous way (i.e. Spring JmsTemplate#receive).
> This goes like so:
> - consumer A fetches 5 items
> - consumer B fetches 20 items
> - consumer A fetches 5 items
> and so on.
>
> The problem we have now is that we are not able to fetch all items from the
> queue - e.g. if there are still 10 items left in the queue and I want to get
> 10, I only get 5 or so.
>
> I've already found the prefetchSize configuration and set it to 0 (also
> tried 1), but still the same problem.
> The problem seems to be that each consumer has an internal 'dispatched
> queue' which already has some entries and consumer A has no possibility to
> access the dispatched queue of consumer B.
>
> For example the web console looks like this:
> Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5;
> Prefetch 0; Max Pending 0
>
> Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue 10;
> Prefetch 0; Max Pending 0
>
> In this case there would be e.g. still 15 items total in the queue, but
> with consumer A I would onyl be able to get 5 items and with consumer B only
> 10 items.
>
> I've tried various configuration options which I found in the ActiveMQ docu
> (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none seemed to
> help.
>
> Can anyone help? Does anyone know how to configure/disable this dispatched
> queue?
>
>
> thanks
> Christoph
>



-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Re: Release date fpr 5.3.1 or 5.4 ( as Re: Problem with 'dispatched queue')

Posted by Gary Tully <ga...@gmail.com>.
Watch the list of outstanding issues for 5.3.1
https://issues.apache.org/activemq/secure/IssueNavigator.jspa?reset=true&mode=hide&sorter/order=DESC&sorter/field=priority&resolution=-1&pid=10520&fixfor=12183

When these are resolved we can do a release.

On 1 February 2010 13:57, Christoph Kutzinski <ku...@gmx.de> wrote:

>
> Which raises my next question:
> When can we expect a release of ActiveMQ 5.3.1 or 5.4?
>
>
> Christoph Kutzinski wrote:
> >
> > I've got the hint that this is related to
> > https://issues.apache.org/activemq/browse/AMQ-2567
> >
> > I've now tried a current Snapshot of ActiveMQ 5.4 which contains the fix
> > for it and it works as expected!
> >
> > Many thanks to Rudolf Janz for pointing me in that direction and to Gary
> > Tully for helping me get the right snapshot release.
> >
> >
> > Christoph
> >
> >
> >
> > Christoph Kutzinski wrote:
> >>
> >> Hi,
> >>
> >> let me first describe my use-case as it is probably a little bit
> >> different from the usual JMS use case:
> >>
> >> We have a single process which fills a JMS queue from a pool of
> available
> >> items (say we have 50 items total).
> >> The we have n consumers (lets say 2) which read the items from the queue
> >> in a synchronous way (i.e. Spring JmsTemplate#receive).
> >> This goes like so:
> >> - consumer A fetches 5 items
> >> - consumer B fetches 20 items
> >> - consumer A fetches 5 items
> >> and so on.
> >>
> >> The problem we have now is that we are not able to fetch all items from
> >> the queue - e.g. if there are still 10 items left in the queue and I
> want
> >> to get 10, I only get 5 or so.
> >>
> >> I've already found the prefetchSize configuration and set it to 0 (also
> >> tried 1), but still the same problem.
> >> The problem seems to be that each consumer has an internal 'dispatched
> >> queue' which already has some entries and consumer A has no possibility
> >> to access the dispatched queue of consumer B.
> >>
> >> For example the web console looks like this:
> >> Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5;
> >> Prefetch 0; Max Pending 0
> >>
> >> Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue
> 10;
> >> Prefetch 0; Max Pending 0
> >>
> >> In this case there would be e.g. still 15 items total in the queue, but
> >> with consumer A I would onyl be able to get 5 items and with consumer B
> >> only 10 items.
> >>
> >> I've tried various configuration options which I found in the ActiveMQ
> >> docu (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none
> seemed
> >> to help.
> >>
> >> Can anyone help? Does anyone know how to configure/disable this
> >> dispatched queue?
> >>
> >>
> >> thanks
> >> Christoph
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27404594.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>


-- 
http://blog.garytully.com

Open Source Integration
http://fusesource.com

Release date fpr 5.3.1 or 5.4 ( as Re: Problem with 'dispatched queue')

Posted by Christoph Kutzinski <ku...@gmx.de>.
Which raises my next question:
When can we expect a release of ActiveMQ 5.3.1 or 5.4?


Christoph Kutzinski wrote:
> 
> I've got the hint that this is related to 
> https://issues.apache.org/activemq/browse/AMQ-2567
> 
> I've now tried a current Snapshot of ActiveMQ 5.4 which contains the fix
> for it and it works as expected!
> 
> Many thanks to Rudolf Janz for pointing me in that direction and to Gary
> Tully for helping me get the right snapshot release.
> 
> 
> Christoph
> 
> 
> 
> Christoph Kutzinski wrote:
>> 
>> Hi,
>> 
>> let me first describe my use-case as it is probably a little bit
>> different from the usual JMS use case:
>> 
>> We have a single process which fills a JMS queue from a pool of available
>> items (say we have 50 items total).
>> The we have n consumers (lets say 2) which read the items from the queue
>> in a synchronous way (i.e. Spring JmsTemplate#receive).
>> This goes like so:
>> - consumer A fetches 5 items
>> - consumer B fetches 20 items
>> - consumer A fetches 5 items
>> and so on.
>> 
>> The problem we have now is that we are not able to fetch all items from
>> the queue - e.g. if there are still 10 items left in the queue and I want
>> to get 10, I only get 5 or so.
>> 
>> I've already found the prefetchSize configuration and set it to 0 (also
>> tried 1), but still the same problem.
>> The problem seems to be that each consumer has an internal 'dispatched
>> queue' which already has some entries and consumer A has no possibility
>> to access the dispatched queue of consumer B.
>> 
>> For example the web console looks like this:
>> Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5;
>> Prefetch 0; Max Pending 0
>> 
>> Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue 10;
>> Prefetch 0; Max Pending 0
>> 
>> In this case there would be e.g. still 15 items total in the queue, but
>> with consumer A I would onyl be able to get 5 items and with consumer B
>> only 10 items.
>> 
>> I've tried various configuration options which I found in the ActiveMQ
>> docu (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none seemed
>> to help.
>> 
>> Can anyone help? Does anyone know how to configure/disable this
>> dispatched queue?
>> 
>> 
>> thanks
>> Christoph
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27404594.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Re: Problem with 'dispatched queue'

Posted by Christoph Kutzinski <ku...@gmx.de>.
I've got the hint that this is related to 
https://issues.apache.org/activemq/browse/AMQ-2567

I've now tried a current Snapshot of ActiveMQ 5.4 which contains the fix for
it and it works as expected!

Many thanks to Rudolf Janz for pointing me in that direction and to Gary
Tully for helping me get the right snapshot release.


Christoph



Christoph Kutzinski wrote:
> 
> Hi,
> 
> let me first describe my use-case as it is probably a little bit different
> from the usual JMS use case:
> 
> We have a single process which fills a JMS queue from a pool of available
> items (say we have 50 items total).
> The we have n consumers (lets say 2) which read the items from the queue
> in a synchronous way (i.e. Spring JmsTemplate#receive).
> This goes like so:
> - consumer A fetches 5 items
> - consumer B fetches 20 items
> - consumer A fetches 5 items
> and so on.
> 
> The problem we have now is that we are not able to fetch all items from
> the queue - e.g. if there are still 10 items left in the queue and I want
> to get 10, I only get 5 or so.
> 
> I've already found the prefetchSize configuration and set it to 0 (also
> tried 1), but still the same problem.
> The problem seems to be that each consumer has an internal 'dispatched
> queue' which already has some entries and consumer A has no possibility to
> access the dispatched queue of consumer B.
> 
> For example the web console looks like this:
> Consumer A; Enqueues 10; Dequeues 5; Dispatched 10; Dispatched Queue 5;
> Prefetch 0; Max Pending 0
> 
> Consumer B; Enqueues 20; Dequeues 10; Dispatched 20; Dispatched Queue 10;
> Prefetch 0; Max Pending 0
> 
> In this case there would be e.g. still 15 items total in the queue, but
> with consumer A I would onyl be able to get 5 items and with consumer B
> only 10 items.
> 
> I've tried various configuration options which I found in the ActiveMQ
> docu (dispatchAsync, alwaysSessionAsync, sendAsync, ...), but none seemed
> to help.
> 
> Can anyone help? Does anyone know how to configure/disable this dispatched
> queue?
> 
> 
> thanks
> Christoph
> 
> 

-- 
View this message in context: http://old.nabble.com/Problem-with-%27dispatched-queue%27-tp27223270p27403534.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.