[DISCUSS] Make system topics non-partitioned by default

[Michael Marshall <mm...@apache.org>]

Hi Pulsar Community,

After several recent issues regarding incorrectly auto created topics,
I think we should default to creating system topics as non-partitioned
topics when auto created.

Lari put together a PR here
https://github.com/apache/pulsar/pull/20397. The PR has relevant
context.

Let me know what you think.

Thanks,
Michael

Re: [DISCUSS] Make system topics non-partitioned by default

[Michael Marshall <mm...@apache.org>]

> Why not disable auto-creation for system topics? and then we can create it before the broker start. which can avoid introduce complex meaning of `allowAutoTopicCreationType` configuration.

A broker cannot know all of the system topics to create at startup.
For example, some system topics are created for each namespace. The
current paradigm implicitly relies on auto creation as a form of
automatic retry to ensure eventual creation of the namespace's system
topic.

> why we chose the non-partitioned topic?

This is likely a deeper question about the role of system topics. I
posit that a system topic should never exceed the throughput
capabilities of a non-partitioned topic. Further, some system topics
implicitly relied on it a single topic to ensure ordering, like the
load balancing topics. This change likely improves stability for those
features.

> maybe we can just create one partition?

This would work, but would only be necessary if we plan to increase
the number of partitions later.

Thanks,
Michael

On Tue, Jun 27, 2023 at 9:12 PM <ma...@gmail.com> wrote:
>
> Hi, Michael
>
> I've got several questions and hope you can help answer.
>
> Why not disable auto-creation for system topics? and then we can create it before the broker start. which can avoid introduce complex meaning of `allowAutoTopicCreationType` configuration.
>
> Plus, why we chose the non-partitioned topic? maybe we can just create one partition? because the non-partitioned topic can't scale by updating the partition number.
>
> Best,
> Mattison
> On 28 Jun 2023 at 06:45 +0800, Michael Marshall <mm...@apache.org>, wrote:
> > Hi Pulsar Community,
> >
> > After several recent issues regarding incorrectly auto created topics,
> > I think we should default to creating system topics as non-partitioned
> > topics when auto created.
> >
> > Lari put together a PR here
> > https://github.com/apache/pulsar/pull/20397. The PR has relevant
> > context.
> >
> > Let me know what you think.
> >
> > Thanks,
> > Michael

Re: [DISCUSS] Make system topics non-partitioned by default

[ma...@gmail.com]

Hi, Michael

I've got several questions and hope you can help answer.

Why not disable auto-creation for system topics? and then we can create it before the broker start. which can avoid introduce complex meaning of `allowAutoTopicCreationType` configuration.

Plus, why we chose the non-partitioned topic? maybe we can just create one partition? because the non-partitioned topic can't scale by updating the partition number.

Best,
Mattison
On 28 Jun 2023 at 06:45 +0800, Michael Marshall <mm...@apache.org>, wrote:
> Hi Pulsar Community,
>
> After several recent issues regarding incorrectly auto created topics,
> I think we should default to creating system topics as non-partitioned
> topics when auto created.
>
> Lari put together a PR here
> https://github.com/apache/pulsar/pull/20397. The PR has relevant
> context.
>
> Let me know what you think.
>
> Thanks,
> Michael