Fair queuing with Artemis

[Stephen Baker <st...@rmssoftwareinc.com>]

 Are there any thoughts on how to achieve fair queuing with Artemis MQ.

We have a limited resource (the consumer) and many of our customers are producing messages for it. We do not want a customer producing a lot of messages from blocking a customer that only sends a few (basically the definition of fair queueing)

On one of our systems we use redis based queuing. There’s a list of currently active customer queues, and the consumer cycles through them pulling off from the respective queue, removing the customer from the list if the queue is empty.

Is it possible to achieve something similar with Artemis?

I would think if it is, it involves auto-create-queues on an existing anycast address, but from there I’m not sure how I would round robin consumption.

If anyone has an idea I’d be interested. If Artemis MQ is the wrong tool for the job, that’s an answer too.

Stephen E. Baker

Re: Fair queuing with Artemis

[Stephen Baker <st...@rmssoftwareinc.com>]

For fair queueing I would need to be able to classify messages as belonging to a particular agent, and then I would need consumers to round robin on the agents with messages in the queue. It’s about not letting one source monopolize the consumers, not ensuring that all the consumers take work.

https://en.wikipedia.org/wiki/Fair_queuing describes the concept and motivation pretty well from a network standpoint, but you can see how the idea translates to any sort of queuing situation with multiple producers.

It’s usually accomplished with a number of individual queues and a shared concept of which queues are in play.

From: Clebert Suconic <cl...@gmail.com>
Date: Saturday, October 22, 2022 at 1:21 AM
To: users@activemq.apache.org <us...@activemq.apache.org>
Subject: Re: Fair queuing with Artemis
You can have multiple consumers on a queue.  That would round robin. You
could disable fetching so you keep messages on the client side.

I’m not sure also If you need a rate limiter or something like that also ?

On Fri, Oct 21, 2022 at 10:35 AM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:

>  Are there any thoughts on how to achieve fair queuing with Artemis MQ.
>
> We have a limited resource (the consumer) and many of our customers are
> producing messages for it. We do not want a customer producing a lot of
> messages from blocking a customer that only sends a few (basically the
> definition of fair queueing)
>
> On one of our systems we use redis based queuing. There’s a list of
> currently active customer queues, and the consumer cycles through them
> pulling off from the respective queue, removing the customer from the list
> if the queue is empty.
>
> Is it possible to achieve something similar with Artemis?
>
> I would think if it is, it involves auto-create-queues on an existing
> anycast address, but from there I’m not sure how I would round robin
> consumption.
>
> If anyone has an idea I’d be interested. If Artemis MQ is the wrong tool
> for the job, that’s an answer too.
>
> Stephen E. Baker
>
--
Clebert Suconic
[EXTERNAL]: This email originated from outside of Rave Mobile Safety. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Re: Fair queuing with Artemis

[Clebert Suconic <cl...@gmail.com>]

You can have multiple consumers on a queue.  That would round robin. You
could disable fetching so you keep messages on the client side.

I’m not sure also If you need a rate limiter or something like that also ?

On Fri, Oct 21, 2022 at 10:35 AM Stephen Baker <
stephen.baker@rmssoftwareinc.com> wrote:

>  Are there any thoughts on how to achieve fair queuing with Artemis MQ.
>
> We have a limited resource (the consumer) and many of our customers are
> producing messages for it. We do not want a customer producing a lot of
> messages from blocking a customer that only sends a few (basically the
> definition of fair queueing)
>
> On one of our systems we use redis based queuing. There’s a list of
> currently active customer queues, and the consumer cycles through them
> pulling off from the respective queue, removing the customer from the list
> if the queue is empty.
>
> Is it possible to achieve something similar with Artemis?
>
> I would think if it is, it involves auto-create-queues on an existing
> anycast address, but from there I’m not sure how I would round robin
> consumption.
>
> If anyone has an idea I’d be interested. If Artemis MQ is the wrong tool
> for the job, that’s an answer too.
>
> Stephen E. Baker
>
-- 
Clebert Suconic