[GitHub] [pulsar] AlecMMiller created a discussion: Failover subscriptions across multiple topics and horizontal scaling

["AlecMMiller (via GitHub)" <gi...@apache.org>]

GitHub user AlecMMiller created a discussion: Failover subscriptions across multiple topics and horizontal scaling

For my use case, I have somewhere on the order of 100 entities, each with the same set of topics associated with them. E.g I'm going to have something like topic A-foo, A-bar, B-foo, B-bar, and so on. I want to have a horizontally scalable number of consumers processing data for these topics, while maintaining ordering consistency for a given entity.

From the documentation, it seems like this should be doable with a failover subscription with multiple topics, as described at

https://pulsar.apache.org/docs/3.1.x/concepts-messaging/#failover--non-partitioned-topics

and this image seems to be pretty much exactly what I want

https://pulsar.apache.org/assets/images/pulsar-failover-subscriptions-3-2b2e00ef5ee525ca7c1fd21dcd67e328.svg

However, the issue I'm running into is that it seems like the first client to subscribe is being assigned all of the topics, instead of having them be reallocated each time a new client subscribes to the subscription

I have tried subscribing with both a regex pattern and with a topic list, and have had no success with either. I've also tried setting the consumer name with a psuedo-random UUID. In all cases, all messages are being routed to the first client to start up. If it matters, I'm using the Node.js client

Am I understanding the way failover subscriptions are supposed to be working correctly, and this should be working? Or is a failover subscription tied to the entire partition that the topics are on, even if there are multiple topics?

GitHub link: https://github.com/apache/pulsar/discussions/20989

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org