ActiveMQ Scheduler - slow enqueue back to th

[Roman P <ro...@checkout.com>]

Hello,

I am using ActiveMQ Scheduler and see a slow delivery rate when trying to
load test the solution.

My scenario: send a batch of scheduled messages (e.g. 1000 messages) with
pretty much the same due to time. (e.g. AMQ_SCHEDULED_DELAY in 20 seconds).
There is one connection and session to serve a producer and one for a
consumer. In this test I expect a similar delivery rate for the producer and
the consumer. But while the producer is fast and can reach 50+
messages/second the consumer consuming rate is around 2-3 messages/second.
As a result, the delivery delay can be 20 + minutes to process several
thousand messages.

When I investigated deeper I realized that the problem is not on the
client-side /consumer. Please note the consumer is fast. 
The problem is that the ActiveMQ Job scheduler re-queues the messages which
are due back to the original queue with the rate of approx 2-3
messages/second. It is even visible when you refresh the Admin UI of Active
MQ. When the messages are re-queued back the consumer can consume them very
quickly.

I haven't found any documentation/details about the scheduler configuration
except a couple of pages about setting delays. Can someone explain why this
happens and a bunch of messages due are hanging in Scheduled state so long?
What is missed?
Please note, I am using the version Active MQ 5.14.3 (docker image)

Thanks,
Roman



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: ActiveMQ Scheduler - slow enqueue back to th

[Tim Bain <tb...@alumni.duke.edu>]

This is the first I've heard of this problem, and I've never used scheduled
messages so I don't have any specific knowledge of the problem. But it
seems like there are two possible reasons for the scheduler to have such
low throughput: 1) the scheduler is configured to only send the scheduled
messages every so often, or 2) the schedule will send as quickly as it can,
but there's a performance problem that prevents it from going faster than
the 2-3 per second you're seeing.

If the latter is the problem, the most beneficial thing would be to attach
JVisualVM to the broker process and capture a CPU Sampling (not Profiling!)
snapshot covering a minute or so of time when the problem is occurring.
With that information, we'd be able to zero in on where the time is being
spent and from there determine how to fix it. Is that something you'd be
able to capture and then share here?

Thanks,
Tim

On Sat, Oct 12, 2019 at 12:53 PM Roman P <ro...@checkout.com> wrote:

> Hello,
>
> I am using ActiveMQ Scheduler and see a slow delivery rate when trying to
> load test the solution.
>
> My scenario: send a batch of scheduled messages (e.g. 1000 messages) with
> pretty much the same due to time. (e.g. AMQ_SCHEDULED_DELAY in 20 seconds).
> There is one connection and session to serve a producer and one for a
> consumer. In this test I expect a similar delivery rate for the producer
> and
> the consumer. But while the producer is fast and can reach 50+
> messages/second the consumer consuming rate is around 2-3 messages/second.
> As a result, the delivery delay can be 20 + minutes to process several
> thousand messages.
>
> When I investigated deeper I realized that the problem is not on the
> client-side /consumer. Please note the consumer is fast.
> The problem is that the ActiveMQ Job scheduler re-queues the messages which
> are due back to the original queue with the rate of approx 2-3
> messages/second. It is even visible when you refresh the Admin UI of Active
> MQ. When the messages are re-queued back the consumer can consume them very
> quickly.
>
> I haven't found any documentation/details about the scheduler configuration
> except a couple of pages about setting delays. Can someone explain why this
> happens and a bunch of messages due are hanging in Scheduled state so long?
> What is missed?
> Please note, I am using the version Active MQ 5.14.3 (docker image)
>
> Thanks,
> Roman
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>