Re: [D] Key_Shared consumer scale behavior with a partitioned topic [pulsar]

["niriven (via GitHub)" <gi...@apache.org>]

GitHub user niriven edited a discussion: Key_Shared consumer scale behavior with a partitioned topic

Hi I am curious what happens in a certain situation -- based on some tests I am running locally. I read through https://pulsar.apache.org/docs/3.3.x/concepts-messaging/#preserving-order-of-processing -- but it does not talk much if at all about partitioned topics with key_shared. In fact there is a note here that states `When the position of the newest message in a topic is X, a key_shared consumer that is newly attached to the same subscription and connected to the topic will not receive any messages until all the messages before X have been acknowledged.` which seems to be inaccurate with partitioned topics and multiple consumers.

Test is as follows
* 1 broker
* 1 persistent topic with two partitions
* One producer application writing to the topic at a rate of 10 sends per second with 10 unique keys, key-N, where N is 0-9
* I have one subscription created ahead of time in Key_Shared mode with `allowOutOfOrderDelivery` = false
* I bring Consumer-1 online, it will process all keys (as expected) and acknowledge but `key-0` messages
  * Consumer-1: `13:45:27.718 [pool-1-thread-1] INFO  c.t.q.p.c.PulsarSpecifiedKeySkipConsumer1 -IS Received: 210, Acknowledged: 198, Skipped: 12, Errored: 0, Keys in 30s [10]: [key-0, key-1, key-2, key-3, key-4, key-5, key-6, key-7, key-8, key-9]`
* I bring Consumer-2 online, Consumer-1 is now only processing 7 of the 10 keys, and Consumer-2 is only processing 1 of the 10 keys
  * Consumer-1: `13:45:51.916 [pool-1-thread-1] INFO  c.t.q.p.c.PulsarSpecifiedKeySkipConsumer - Received: 380, Acknowledged: 368, Skipped: 12, Errored: 0, Keys in 30s [7]: [key-1, key-2, key-3, key-4, key-5, key-6, key-7]`
  * Consumer-2: `13:45:49.315 [pool-1-thread-1] INFO  c.t.q.p.c.PulsarSpecifiedKeySkipConsumer - Received: 30, Acknowledged: 30, Skipped: 0, Errored: 0, Keys in 30s [1]: [key-9]`

Based on the test, `key-8` remains unaccounted during this period I do not ack the `key-0` messages. So based on this test I have a few questions
* I assume when I ack all of the `key-0` messages it will re-balance appropriately?
* What is happening here during this time with `key-8` not being delivered to any consumer?
* Is there any documentation on how key_shared works with partitioned topics and consumer scale up / down?

NOTE: When I set `allowOutOfOrderDelivery` to false, it distributes as I'd expect (All consumers are working on all keys during scale up / down)

GitHub link: https://github.com/apache/pulsar/discussions/22912

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org