[GitHub] [pulsar] yehancha created a discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

[GitBox <gi...@apache.org>]

GitHub user yehancha created a discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

My requirement is to create 200K topics and get 100K msg/s throughput across all those topics. (Each topic gets 1 msg every 2 seconds.)

I created 80K topics under 8 namespaces and got 40K msg/s. But when I try to go beyond that, the throughput drops.

I have 3 x 64GB bookkeepers. Currently, I have 2 x 120GB brokers but I also have tested the setup with number of different configurations for the broker (memory per broker and number of brokers) as well.

The above limitation seems to be with bookkeeper usage. Because, when I use `non-persistent` topics, I could reach up to 150K topics and got 75K msg/s only with 1 x 120GB broker. Did not test beyond that.

I appreciate any advice, suggestion on;
- how should I configure this setup (number of bookkeepers and brokers, there memory and CPU requirements, etc.)
- where should I look for the bottlenecks
- what configurations I should tweak in bookkeeper and broker to get more throughput
- anything else

GitHub link: https://github.com/apache/pulsar/discussions/19862

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] yehancha added a comment to the discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

[GitBox <gi...@apache.org>]

GitHub user yehancha added a comment to the discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

The issue seems to be with `DispatchRate` limit. In my case, I need 0.5 b/s throughput on a single topic. I have set the dispatch rate limit to 4 kb/s, well above my need. But even with that, the throughput slows down (which I will look into in the future). When I remove this limit, everything works fine.

I will keep the discussion open until I hit my initial target of 100K msg/s on 200K topics to discuss any other issues on the way.

GitHub link: https://github.com/apache/pulsar/discussions/19862#discussioncomment-5378399

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org

[GitHub] [pulsar] yehancha edited a discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

[GitBox <gi...@apache.org>]

GitHub user yehancha edited a discussion: How to tweak a Pulsar cluster to have a large number of topics with a reasonable throughput

My requirement is to create 200K topics and get 100K msg/s throughput across all those topics. (Each topic gets 1 msg every 2 seconds and these are 1KB messages.)

I created 80K topics under 8 namespaces and got 40K msg/s. But when I try to go beyond that, the throughput drops.

I have 3 x 64GB bookkeepers. Currently, I have 2 x 120GB brokers but I also have tested the setup with number of different configurations for the broker (memory per broker and number of brokers) as well.

The above limitation seems to be with bookkeeper usage. Because, when I use `non-persistent` topics, I could reach up to 150K topics and got 75K msg/s only with 1 x 120GB broker. Did not test beyond that.

I appreciate any advice, suggestion on;
- how should I configure this setup (number of bookkeepers and brokers, there memory and CPU requirements, etc.)
- where should I look for the bottlenecks
- what configurations I should tweak in bookkeeper and broker to get more throughput
- anything else

GitHub link: https://github.com/apache/pulsar/discussions/19862

----
This is an automatically sent email for commits@pulsar.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@pulsar.apache.org