[GitHub] [pulsar] Samreay created a discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

[GitBox <gi...@apache.org>]

GitHub user Samreay created a discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

Hi all! We're looking at adding Pulsar to our tech stack, and we're trying to figure out what we can do, and more importantly, what we can't yet do.

The first thing, which I believe is just not possible due to the nature of the tech, are priority messages and priority consumers, with failover subscriptions being a way to get fairly close to a priority consumer.

More importantly than this, in RabbitMQ with a group of consumers pulling from the same queue, you can easily set it up so that a consumer which has not ack'd the previous message isn't sent new messages. This is handy for us as an event driven work queue, because most jobs takes about 10ms, but some will take around 10s. This functionality helps things not pile up on the worker unlucky enough to be allocated the longer processing job.

Is this functionality built into the [shared subscription](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#shared) at all, and if not, would anyone have any recommendations about how to approach the problem?

GitHub link: https://github.com/apache/pulsar/discussions/20211

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] merlimat added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

[GitBox <gi...@apache.org>]

GitHub user merlimat added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

> The first thing, which I believe is just not possible due to the nature of the tech, are priority messages and priority consumers, with failover subscriptions being a way to get fairly close to a priority consumer.

For messages priority, you can use separate topics for each priority.

Consumer priority can be set for shared subscriptions: https://pulsar.apache.org/api/client/2.11.x/org/apache/pulsar/client/api/ConsumerBuilder.html#priorityLevel(int) 

> Is this functionality built into the [shared subscription](https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#shared) at all, and if not, would anyone have any recommendations about how to approach the problem? 

You can completely disable prefetching by setting the consumer receiver queue to 0: each consumer will only have the message that is currently process and won't block any other message.

GitHub link: https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776848

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] Samreay added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

[GitBox <gi...@apache.org>]

GitHub user Samreay added a comment to the discussion: Migrating from RabbitMQ: shared consumers with new messages only on ack

Thanks for the quick response, I appreciate it!

Ah `priorityLevel` looks interesting. Is that something only available in the Java client? I can't see it in the [feature matrix](https://docs.google.com/spreadsheets/d/1YHYTkIXR8-Ql103u-IMI18TXLlGStK8uJjDsOOA0T20/edit#gid=1784579914), but a search through the [python docs](https://pulsar.apache.org/api/python/3.1.x/pulsar.Client.html#subscribe) doesn't show up anything about priority. Have I misunderstood something, or is this a feature yet to make it across?

> You can completely disable prefetching by setting the consumer receiver queue to 0: each consumer will only have the message that is currently process and won't block any other message.

That's also good to know, and I see the `client.subscribe` function from above does indeed have a `receiver_queue_size` kwarg to make things simple. I'll give it a whirl this afternoon and see if I can get it all working and report back!

GitHub link: https://github.com/apache/pulsar/discussions/20211#discussioncomment-5776889

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org