[Question] Limit on Number of Topics Per Namespace

[Michael Marshall <mi...@gmail.com>]

Hi All,

Recently in the pulsar slack channel, someone asked about the scalability
(and limits) of the number of topics in a pulsar cluster. Addison Higham
responded with the following:

> For numbers of topics, there is a few notes:
> - you will want to spread these topics over a sufficient number of
namespaces. We typically recommended about 10k topics at most per namespace
to handle some metadata scalability in Pulsar, but once you spread across
namespaces, it is fine. For the use case you mentioned, you might do that
by hashing/prefix mapping to a number of namespaces
> - with that number of topics, you typically will need more memory per
broker and more brokers, but you can scale out quite well to add more topics
> - You will want to look into adding more "bundles" in each namespace.

I had not previously seen any information on recommended topic limits per
namespace, and I found the recommended 10k limit lower than I would have
expected given that pulsar is stated to scale to 1 million topics but
doesn't mention a limit on number of topics per namespace in the
documentation.

Does anyone have more information about this limit? What happens when a
namespace exceeds the limit? What is the main bottleneck leading to this
limit? I don't believe that pulsar documentation explicitly makes
recommendations on these limits or on the trade offs associated with
exceeding them.

Additionally, is there a limit on the number of topics per broker?

If we don't already have these answers documented in the pulsar
documentation, which I don't think we do, I think it'd be valuable to add a
section to help users understand how to scale out the number of topics.

Thanks,
Michael Marshall

Re: [Question] Limit on Number of Topics Per Namespace

[Michael Marshall <mi...@gmail.com>]

Thank you for your response Sijie! It provides helpful context.

On Tue, Apr 6, 2021 at 10:31 AM Sijie Guo <gu...@gmail.com> wrote:

> On Mon, Apr 5, 2021 at 9:57 PM Michael Marshall <mi...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > Recently in the pulsar slack channel, someone asked about the scalability
> > (and limits) of the number of topics in a pulsar cluster. Addison Higham
> > responded with the following:
> >
> > > For numbers of topics, there is a few notes:
> > > - you will want to spread these topics over a sufficient number of
> > namespaces. We typically recommended about 10k topics at most per
> namespace
> > to handle some metadata scalability in Pulsar, but once you spread across
> > namespaces, it is fine. For the use case you mentioned, you might do that
> > by hashing/prefix mapping to a number of namespaces
> > > - with that number of topics, you typically will need more memory per
> > broker and more brokers, but you can scale out quite well to add more
> > topics
> > > - You will want to look into adding more "bundles" in each namespace.
> >
> > I had not previously seen any information on recommended topic limits per
> > namespace, and I found the recommended 10k limit lower than I would have
> > expected given that pulsar is stated to scale to 1 million topics but
> > doesn't mention a limit on number of topics per namespace in the
> > documentation.
> >
> > Does anyone have more information about this limit? What happens when a
> > namespace exceeds the limit? What is the main bottleneck leading to this
> > limit? I don't believe that pulsar documentation explicitly makes
> > recommendations on these limits or on the trade offs associated with
> > exceeding them.
> >
>
> The limitation is coming from zookeeper. ZooKeeper doesn't support
> iterating the children under one znode.
> So it has to fetch the entire children. If you put too many topics under
> one namespace, it will break the list-topics action.
>
>
> >
> > Additionally, is there a limit on the number of topics per broker?
> >
>
> The number of topics a broker can support is majorly dependent on the
> memory of a broker.
>
>
> >
> > If we don't already have these answers documented in the pulsar
> > documentation, which I don't think we do, I think it'd be valuable to
> add a
> > section to help users understand how to scale out the number of topics.
> >
>
> Yes. We should consider adding a section for it.
>
>
> >
> > Thanks,
> > Michael Marshall
> >
>

Re: [Question] Limit on Number of Topics Per Namespace

[Sijie Guo <gu...@gmail.com>]

On Mon, Apr 5, 2021 at 9:57 PM Michael Marshall <mi...@gmail.com>
wrote:

> Hi All,
>
> Recently in the pulsar slack channel, someone asked about the scalability
> (and limits) of the number of topics in a pulsar cluster. Addison Higham
> responded with the following:
>
> > For numbers of topics, there is a few notes:
> > - you will want to spread these topics over a sufficient number of
> namespaces. We typically recommended about 10k topics at most per namespace
> to handle some metadata scalability in Pulsar, but once you spread across
> namespaces, it is fine. For the use case you mentioned, you might do that
> by hashing/prefix mapping to a number of namespaces
> > - with that number of topics, you typically will need more memory per
> broker and more brokers, but you can scale out quite well to add more
> topics
> > - You will want to look into adding more "bundles" in each namespace.
>
> I had not previously seen any information on recommended topic limits per
> namespace, and I found the recommended 10k limit lower than I would have
> expected given that pulsar is stated to scale to 1 million topics but
> doesn't mention a limit on number of topics per namespace in the
> documentation.
>
> Does anyone have more information about this limit? What happens when a
> namespace exceeds the limit? What is the main bottleneck leading to this
> limit? I don't believe that pulsar documentation explicitly makes
> recommendations on these limits or on the trade offs associated with
> exceeding them.
>

The limitation is coming from zookeeper. ZooKeeper doesn't support
iterating the children under one znode.
So it has to fetch the entire children. If you put too many topics under
one namespace, it will break the list-topics action.


>
> Additionally, is there a limit on the number of topics per broker?
>

The number of topics a broker can support is majorly dependent on the
memory of a broker.


>
> If we don't already have these answers documented in the pulsar
> documentation, which I don't think we do, I think it'd be valuable to add a
> section to help users understand how to scale out the number of topics.
>

Yes. We should consider adding a section for it.


>
> Thanks,
> Michael Marshall
>