[GitHub] [pulsar] ramprasadp created a discussion: how can I dynamically increase consumers

[GitBox <gi...@apache.org>]

GitHub user ramprasadp created a discussion: how can I dynamically increase consumers

I am currently using kafka  , sorry no experience of pulsar
Our  production load is very peaky and distributed across multiple topics . 
Number of topics I restrict to "n" , usually 100 ,  because kafka does not support too many topics. And fixed 12 partitions 

The biggest challenge I see is that suddenly when the load ( number of events ) entering a single topic increases There is no way I can increase the number consumers for just that topic beyond 12 
The only way to scale seems to increase number of partitions which is not possible in runtime. Also the consumers are sometimes unevenly loaded and rebalancing kills the performance , for a good 5-10 minutes the consumers go starved when that h

Does Pulsar address these problems 


GitHub link: https://github.com/apache/pulsar/discussions/19067

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] cbornet added a comment to the discussion: how can I dynamically increase consumers

[GitBox <gi...@apache.org>]

GitHub user cbornet added a comment to the discussion: how can I dynamically increase consumers

Yes, Pulsar addresses these problems by having more ways to consume the data.
If you use `Shared` or `KeyShared` subscriptions, you're not limited in the number of consumers that consume from the topic.

GitHub link: https://github.com/apache/pulsar/discussions/19067#discussioncomment-4854405

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org