Blog post series: Rearchitecting Apache Pulsar to handle 100 million topics, Part 1

[Lari Hotari <lh...@apache.org>]

Hi all,

I blogged about rearchitecting Apache Pulsar to handle 100 million topics:
https://codingthestreams.com/pulsar/2022/10/21/possible-high-level-architecture.html
This is the latest blog post in a long series that I have been writing. 

It's related to the thread "Planning for Apache Pulsar 3.0", https://lists.apache.org/thread/1bofpck07fgnv118s2z9qtpz7tvd8fg9 . 

I feel that with the blog post series I have a better chance to communicate what I have in mind and go deeper in details so that you could also learn about the possibilities there are in front of us.

I'm continuing the blog post series with more details and blogs about the PoC to reach the goal of 100 million topics. Another goal is to address Pulsar load balancing problems which cause availability issues. These problems are stated in the blog post series. I'm sure, that I have made mistakes in the posts. I hope that you can read through those and focus on what is the main point, making Pulsar even better. 

I'm looking forward to your feedback, suggestions and questions, for improving the current redesign proposal. This is only a seed and for it to grow, it needs more care from the broader Apache Pulsar community. There will be more opportunities to contribute once this proceeds. The best way to contribute at this stage is to provide feedback and express willingness to join PoC implementation.

The most significant change in the architecture redesign is removing the existing "namespace bundle" concept. I believe that it is evident, and the sooner we make the change, the quicker it will be to reach the goals that we all have for improving Pulsar. The PoC will tell whether these assumptions are true or not.

I'm excited about this challenge. I'm sure it will be a learning experience, even if it wouldn't succeed. Since this is a PoC, it can be made fun too. Let's enjoy open source as it's best and make this happen. 100 million topics in Apache Pulsar. Do we have to stop there? :) 

Best Regards,

-Lari