Re: Realistic maximum number of topics in production

[frank <fr...@frankkelly.us>]

Hey Johnny,

   Sorry for the delay on replying.  You might find this talk I did of use https://youtu.be/vE7aKZT9r7k



TL;DR you will need to scale up ZooKeeper (CPU, Memory, Disk) and Brokers (Memory for sure).

There's probably more I've forgotten that you'll find in the talk.



In the end we would be creating 500k+ topics per day and since tiered storage would still require ZooKeeper to keep track of these topics so we created our own "offloader" to read topics once they are finished and writes the data to S3. 



-Frank









---- On Thu, 24 Feb 2022 05:08:33 -0500 Johnny Miller <jo...@digitalis.io> wrote ----



Hi - I have a project I am at the design phase in, and it would be interesting to leverage the million topic scale in Pulsar. I have experienced problems in similar tech where these limits are more"hypothetical" rather than real.



Is this realistic in prod/operationally? Not necessarily a lot of data, more just lots of topics.



Any real stories, experiences or war wounds around this level of scale would be appreciated.



Thanks,



Johnny















--

The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. http://www.digitalis.io

Re: Realistic maximum number of topics in production

[Tecno Brain <ce...@gmail.com>]

Thanks a lot Frank!


On Thu, Mar 10, 2022 at 6:08 AM frank <fr...@frankkelly.us> wrote:

> Hey Johnny,
>    Sorry for the delay on replying.  You might find this talk I did of use
> https://youtu.be/vE7aKZT9r7k
>
> TL;DR you will need to scale up ZooKeeper (CPU, Memory, Disk) and Brokers
> (Memory for sure).
> There's probably more I've forgotten that you'll find in the talk.
>
> In the end we would be creating 500k+ topics per day and since tiered
> storage would still require ZooKeeper to keep track of these topics so we
> created our own "offloader" to read topics once they are finished and
> writes the data to S3.
>
> -Frank
>
>
>
>
> ---- On Thu, 24 Feb 2022 05:08:33 -0500 *Johnny Miller
> <johnny@digitalis.io <jo...@digitalis.io>>* wrote ----
>
> Hi - I have a project I am at the design phase in, and it would be
> interesting to leverage the million topic scale in Pulsar. I have
> experienced problems in similar tech where these limits are
> more"hypothetical" rather than real.
>
> Is this realistic in prod/operationally? Not necessarily a lot of data,
> more just lots of topics.
>
> Any real stories, experiences or war wounds around this level of scale
> would be appreciated.
>
> Thanks,
>
> Johnny
>
>
> --
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments. WARNING: Computer viruses can be
> transmitted via email. The recipient should check this email and any
> attachments for the presence of viruses. The company accepts no liability
> for any damage caused by any virus transmitted by this email.
> www.digitalis.io
>
>
>
>