You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Michael Marshall <mm...@apache.org> on 2023/06/01 04:46:46 UTC
Re: [DISCUSS] PIP-263: Just auto-create no-partitioned DLQ And Prevent auto-create a DLQ for a DLQ
We've had several bugs related to this topic of auto created
partitioned topics recently. These three PRs come to mind:
* https://github.com/apache/pulsar/pull/20370
* https://github.com/apache/pulsar/pull/20392
* https://github.com/apache/pulsar/pull/20397
Those PRs rely on server side inference based on the topic name to
know that a topic should not be a partitioned topic.
In general, these problems hint to me that we need better definitions
for "system" and "special" topics that break the rules of auto created
topics in well defined ways.
I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead.
> I agree with you, but now the problem is that we still need
> an identifier to say that it is a DLQ. Do you have some
> suggestions?
The identifier is the name. The topics have `-DLQ` and `-RETRY` as
suffixes, which makes them "special" (which reminds me of this thread
[0]). We could choose to always make these non partitioned or to make
them configurable in some other way.
Or, we could consider giving producers and consumers the option to
configure a `partitionCountHint`. This could be used by the broker
during auto creation to create a partitioned topic with x partitions
when greater than 0 and to create a non-partitioned topic when 0. This
moves the logic out of inference on topic names and into the realm of
client configuration. However, it could create hard to debug
scenarios.
Thanks,
Michael
[0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w
On Wed, May 31, 2023 at 7:42 AM Yubiao Feng
<yu...@streamnative.io.invalid> wrote:
>
> Hi @Enrico
>
> > I think that it is better to add flags like:
> > - allowAutoTopiCreation: default "true", if "false" the broker
> > won't create the topic in any case
> > autoTopicCreationMode: undefined/partitioned/non-partitioned
>
> I agree with you, but now the problem is that we still need
> an identifier to say that it is a DLQ. Do you have some
> suggestions?
>
> Thanks
>
> Yubiao Feng
>
>
> On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Yubiao,
> > I understand your problem: DLQ and retry topic don't play well with
> > automatic topic creation.
> >
> > We are introducing wire protocol changes and also starting to add
> > topic properties for every new topic (they come with a cost on
> > metadata).
> >
> > I am not sure that adding the purposeOfAutoCreatedTopic property adds
> > much value and I suggest to drop it.
> > Logging on the broker that you created the topic is enough. I don't
> > think that any application or even the broker would use that piece of
> > information.
> >
> >
> > Regarding topicPurpose I think that having this "list" of
> > possibilities will become hard to maintain un the future.
> >
> > I think that it is better to add flags like:
> > - allowAutoTopiCreation: default "true", if "false" the broker won't
> > create the topic in any case
> > - autoTopicCreationMode: undefined/partitioned/non-partitioned
> >
> > One problem with partitioned topic and DLQ is that you probably want
> > the same number of partitions on the side topic as the original topic,
> > so probably you have to carry that parameter (number of partitions).
> >
> > IIRC the Subscribe command is sent directly per each partition, so I
> > don't know how these things may work well together.
> >
> > Enrico
> >
> >
> > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng
> > <xi...@apache.org> ha scritto:
> > >
> > > Hi Yubiao,
> > >
> > > Appreciate your effort in initiating this PIP. I believe these changes
> > will
> > > address the existing issues and make DLQ and Retry Topic handling more
> > > efficient and straightforward.
> > >
> > > The goals you outlined are clear and, upon implementation, will improve
> > the
> > > overall functionality of Pulsar. The proposed API changes also seem
> > > suitable for achieving the desired outcomes.
> > >
> > > Looking forward to the progress on this PIP.
> > >
> > > Best regards,
> > > Xiangying
> > >
> > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng
> > > <yu...@streamnative.io.invalid> wrote:
> > >
> > > > Hi community
> > > >
> > > > I started a PIP about "Just auto-create no-partitioned DLQ And Prevent
> > > > auto-create a DLQ for a DLQ".
> > > >
> > > > PIP link: https://github.com/apache/pulsar/issues/20033
> > > >
> > > > ### Motivation
> > > >
> > > > #### Just auto-create no-partitioned DLQ/Retry Topic
> > > > If enabled the config `allowAutoTopicCreation,` Pulsar will
> > auto-create a
> > > > topic when the client loads it; After setting config
> > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`,
> > Pulsar
> > > > will auto-create a partitioned topic(which have two partitions) when
> > the
> > > > client loads it.
> > > >
> > > > After the above, if using the feature [Retry Topic](
> > > >
> > > >
> > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic
> > > > )
> > > > and [DLQ](
> > > >
> > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic
> > > > )
> > > > enable topic auto-creation, we will get a partitioned DLQ and a
> > partitioned
> > > > Retry Topic like this:
> > > > - `{primary_topic_name}-{sub_name}-DLQ`
> > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0`
> > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1`
> > > > - `{primary_topic_name}-{sub_name}-RETRY`
> > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0`
> > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1`
> > > >
> > > > ----
> > > >
> > > > I feel that almost all users will not use the multi-partitioned DLQ or
> > > > multi-partitioned Retry topic because there is a bug that causes the
> > above
> > > > behavior to be incorrect, but we have yet to receive any issues about
> > it.
> > > > This bug causes the above behavior to look like this: When the
> > partitioned
> > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will
> > create a
> > > > partitioned topic meta which has two partitioned but only create a
> > topic
> > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic named
> > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at this
> > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed bug
> > > > description.
> > > >
> > > > So I want to change the behavior to Just auto-create no-partitioned
> > > > DLQ/Retry Topic.
> > > >
> > > > ----
> > > >
> > > > #### Prevent auto-create the DLQ for a DLQ
> > > > Please look at this [Discussion](
> > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) for
> > the
> > > > detail.
> > > >
> > > > ----
> > > >
> > > > ### Goal
> > > >
> > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other words:
> > > > prevent auto-create partitioned DLQ)
> > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic
> > > > - roles:
> > > > - DLQ will not auto-create for a DLQ
> > > > - Retry Topic will not auto-create for a Retry Topic
> > > > - DLQ will not auto-create for a Retry Topic
> > > > - Retry Topic will not auto-create for a DLQ
> > > > - client changes: Clients will not create a DLQ for a DLQ
> > > > - broker changes: rejected the request which wants to auto-create a
> > DLQ
> > > > for a DLQ
> > > >
> > > > ----
> > > >
> > > > ### API Changes
> > > >
> > > > #### CommandSubscribe.java
> > > > ```java
> > > > /**
> > > > * This is an enumeration value with tree options: "standard", "dead
> > > > letter", "retry letter".
> > > > */
> > > > private String topicPurpose;
> > > > ```
> > > >
> > > > #### Properties of Topic
> > > > ```properties
> > > > "purposeOfAutoCreatedTopic": value with tree options: "standard", "dead
> > > > letter", "retry letter"
> > > > ```
> > > >
> > > > Why not use two properties: `isAutoCreated` and `topicPurpose`?
> > > > Because there is a scenario like this: auto-create a topic, use it as
> > a DLQ
> > > > after a few days, and not use it as a DLQ after a few days, this Topic
> > will
> > > > be allowed to have DLQ/Retry Topic. We only mark the topics created for
> > > > DLQ/Retry purposes.
> > > >
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> >
Re: [DISCUSS] PIP-263: Just auto-create no-partitioned DLQ And Prevent auto-create a DLQ for a DLQ
Posted by Asaf Mesika <as...@gmail.com>.
I agree with you. wasAutoCreated seems a bit like a patch in light of the
recent issues.
On Fri, Jun 9, 2023 at 9:03 AM Michael Marshall <mm...@apache.org>
wrote:
> > In general, these problems hint to me that we need better definitions
> > for "system" and "special" topics that break the rules of auto created
> > topics in well defined ways.
>
> This point reminds me of this discussion from a year and a half ago:
>
> https://lists.apache.org/thread/qgbpzr6o3k5rqbs2jvpkdh8hr9jpnw39
>
> For PIP 124, I challenged a feature based on the question of whether
> the DLQ is a special topic or not.
>
> Ultimately, we determined it is a special topic (see Matteo's response
> later in that thread).
>
> I think it is interesting because it feels like we continue to find
> exceptions.
>
> Just this week, we found yet another and lamented the lack of design
> for these topics:
> https://github.com/apache/pulsar/pull/20514#issuecomment-1579937937.
>
> I mention those points because I am concerned that creating a DLQ
> specific piece of boolean metadata in the topic properties does not
> move us toward a unified and extensible abstraction.
>
> Unless we want to continue to infer everything from topic names, it
> seems like we almost need a "topic type" enum to capture the many
> types of special topics we have. Type has the same issues I mentioned
> in my last email, but it could help create a more generic solution.
>
> > ### Properties of Topic
> > ```properties
> > "wasAutoCreated": boolean
> > ```
>
> I don't have the answer yet, but I think we should spend more time
> working on a general solution for topic classification before adding
> per topic metadata.
>
> I'd be interested to know what others think on the subject.
>
> Thanks,
> Michael
>
> On Thu, Jun 1, 2023 at 2:08 AM Yubiao Feng
>
> <yu...@streamnative.io.invalid> wrote:
> >
> > Hi Michael, Enrico
> >
> > > The identifier is the name. The topics have `-DLQ` and `-RETRY`
> > > as suffixes, which makes them "special" (which reminds me of this
> > > thread [0]). We could choose to always make these non partitioned
> > > or to make them configurable in some other way.
> >
> > I changed the design like this:
> >
> > ### Properties of Topic
> > ```properties
> > "wasAutoCreated": boolean
> > ```
> >
> > - If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned
> > topics will be automatically created
> > - If the property `wasAutoCreated` of the topic is true and the topic
> name
> > ends with "-RETRY" or "-DLQ", retry topics and dead letter queues
> > will no longer be created for this topic.
> >
> > How are you feeling?
> >
> > Thanks
> > Yubiao Feng
> >
> > On Thu, Jun 1, 2023 at 12:48 PM Michael Marshall <mm...@apache.org>
> > wrote:
> >
> > > We've had several bugs related to this topic of auto created
> > > partitioned topics recently. These three PRs come to mind:
> > >
> > > * https://github.com/apache/pulsar/pull/20370
> > > * https://github.com/apache/pulsar/pull/20392
> > > * https://github.com/apache/pulsar/pull/20397
> > >
> > > Those PRs rely on server side inference based on the topic name to
> > > know that a topic should not be a partitioned topic.
> > >
> > > In general, these problems hint to me that we need better definitions
> > > for "system" and "special" topics that break the rules of auto created
> > > topics in well defined ways.
> > >
> > > I agree with Enrico that "purposeOfAutoCreatedTopic" will create
> overhead.
> > >
> > > > I agree with you, but now the problem is that we still need
> > > > an identifier to say that it is a DLQ. Do you have some
> > > > suggestions?
> > >
> > > The identifier is the name. The topics have `-DLQ` and `-RETRY` as
> > > suffixes, which makes them "special" (which reminds me of this thread
> > > [0]). We could choose to always make these non partitioned or to make
> > > them configurable in some other way.
> > >
> > > Or, we could consider giving producers and consumers the option to
> > > configure a `partitionCountHint`. This could be used by the broker
> > > during auto creation to create a partitioned topic with x partitions
> > > when greater than 0 and to create a non-partitioned topic when 0. This
> > > moves the logic out of inference on topic names and into the realm of
> > > client configuration. However, it could create hard to debug
> > > scenarios.
> > >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w
> > >
> > > On Wed, May 31, 2023 at 7:42 AM Yubiao Feng
> > > <yu...@streamnative.io.invalid> wrote:
> > > >
> > > > Hi @Enrico
> > > >
> > > > > I think that it is better to add flags like:
> > > > > - allowAutoTopiCreation: default "true", if "false" the broker
> > > > > won't create the topic in any case
> > > > > autoTopicCreationMode: undefined/partitioned/non-partitioned
> > > >
> > > > I agree with you, but now the problem is that we still need
> > > > an identifier to say that it is a DLQ. Do you have some
> > > > suggestions?
> > > >
> > > > Thanks
> > > >
> > > > Yubiao Feng
> > > >
> > > >
> > > > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eolivelli@gmail.com
> >
> > > wrote:
> > > >
> > > > > Yubiao,
> > > > > I understand your problem: DLQ and retry topic don't play well
> with
> > > > > automatic topic creation.
> > > > >
> > > > > We are introducing wire protocol changes and also starting to add
> > > > > topic properties for every new topic (they come with a cost on
> > > > > metadata).
> > > > >
> > > > > I am not sure that adding the purposeOfAutoCreatedTopic property
> adds
> > > > > much value and I suggest to drop it.
> > > > > Logging on the broker that you created the topic is enough. I don't
> > > > > think that any application or even the broker would use that piece
> of
> > > > > information.
> > > > >
> > > > >
> > > > > Regarding topicPurpose I think that having this "list" of
> > > > > possibilities will become hard to maintain un the future.
> > > > >
> > > > > I think that it is better to add flags like:
> > > > > - allowAutoTopiCreation: default "true", if "false" the broker
> won't
> > > > > create the topic in any case
> > > > > - autoTopicCreationMode: undefined/partitioned/non-partitioned
> > > > >
> > > > > One problem with partitioned topic and DLQ is that you probably
> want
> > > > > the same number of partitions on the side topic as the original
> topic,
> > > > > so probably you have to carry that parameter (number of
> partitions).
> > > > >
> > > > > IIRC the Subscribe command is sent directly per each partition, so
> I
> > > > > don't know how these things may work well together.
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng
> > > > > <xi...@apache.org> ha scritto:
> > > > > >
> > > > > > Hi Yubiao,
> > > > > >
> > > > > > Appreciate your effort in initiating this PIP. I believe these
> > > changes
> > > > > will
> > > > > > address the existing issues and make DLQ and Retry Topic handling
> > > more
> > > > > > efficient and straightforward.
> > > > > >
> > > > > > The goals you outlined are clear and, upon implementation, will
> > > improve
> > > > > the
> > > > > > overall functionality of Pulsar. The proposed API changes also
> seem
> > > > > > suitable for achieving the desired outcomes.
> > > > > >
> > > > > > Looking forward to the progress on this PIP.
> > > > > >
> > > > > > Best regards,
> > > > > > Xiangying
> > > > > >
> > > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng
> > > > > > <yu...@streamnative.io.invalid> wrote:
> > > > > >
> > > > > > > Hi community
> > > > > > >
> > > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And
> > > Prevent
> > > > > > > auto-create a DLQ for a DLQ".
> > > > > > >
> > > > > > > PIP link: https://github.com/apache/pulsar/issues/20033
> > > > > > >
> > > > > > > ### Motivation
> > > > > > >
> > > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic
> > > > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will
> > > > > auto-create a
> > > > > > > topic when the client loads it; After setting config
> > > > > > > `allowAutoTopicCreationType=partitioned,
> defaultNumPartitions=2`,
> > > > > Pulsar
> > > > > > > will auto-create a partitioned topic(which have two partitions)
> > > when
> > > > > the
> > > > > > > client loads it.
> > > > > > >
> > > > > > > After the above, if using the feature [Retry Topic](
> > > > > > >
> > > > > > >
> > > > >
> > >
> https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic
> > > > > > > )
> > > > > > > and [DLQ](
> > > > > > >
> > > > >
> > >
> https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic
> > > > > > > )
> > > > > > > enable topic auto-creation, we will get a partitioned DLQ and a
> > > > > partitioned
> > > > > > > Retry Topic like this:
> > > > > > > - `{primary_topic_name}-{sub_name}-DLQ`
> > > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0`
> > > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1`
> > > > > > > - `{primary_topic_name}-{sub_name}-RETRY`
> > > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0`
> > > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1`
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > I feel that almost all users will not use the multi-partitioned
> > > DLQ or
> > > > > > > multi-partitioned Retry topic because there is a bug that
> causes
> > > the
> > > > > above
> > > > > > > behavior to be incorrect, but we have yet to receive any issues
> > > about
> > > > > it.
> > > > > > > This bug causes the above behavior to look like this: When the
> > > > > partitioned
> > > > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar
> will
> > > > > create a
> > > > > > > partitioned topic meta which has two partitioned but only
> create a
> > > > > topic
> > > > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic
> > > named
> > > > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look
> at
> > > this
> > > > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a
> detailed
> > > bug
> > > > > > > description.
> > > > > > >
> > > > > > > So I want to change the behavior to Just auto-create
> no-partitioned
> > > > > > > DLQ/Retry Topic.
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > #### Prevent auto-create the DLQ for a DLQ
> > > > > > > Please look at this [Discussion](
> > > > > > >
> https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8)
> > > for
> > > > > the
> > > > > > > detail.
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > ### Goal
> > > > > > >
> > > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the
> other
> > > words:
> > > > > > > prevent auto-create partitioned DLQ)
> > > > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic
> > > > > > > - roles:
> > > > > > > - DLQ will not auto-create for a DLQ
> > > > > > > - Retry Topic will not auto-create for a Retry Topic
> > > > > > > - DLQ will not auto-create for a Retry Topic
> > > > > > > - Retry Topic will not auto-create for a DLQ
> > > > > > > - client changes: Clients will not create a DLQ for a DLQ
> > > > > > > - broker changes: rejected the request which wants to
> > > auto-create a
> > > > > DLQ
> > > > > > > for a DLQ
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > ### API Changes
> > > > > > >
> > > > > > > #### CommandSubscribe.java
> > > > > > > ```java
> > > > > > > /**
> > > > > > > * This is an enumeration value with tree options: "standard",
> > > "dead
> > > > > > > letter", "retry letter".
> > > > > > > */
> > > > > > > private String topicPurpose;
> > > > > > > ```
> > > > > > >
> > > > > > > #### Properties of Topic
> > > > > > > ```properties
> > > > > > > "purposeOfAutoCreatedTopic": value with tree options:
> "standard",
> > > "dead
> > > > > > > letter", "retry letter"
> > > > > > > ```
> > > > > > >
> > > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`?
> > > > > > > Because there is a scenario like this: auto-create a topic,
> use it
> > > as
> > > > > a DLQ
> > > > > > > after a few days, and not use it as a DLQ after a few days,
> this
> > > Topic
> > > > > will
> > > > > > > be allowed to have DLQ/Retry Topic. We only mark the topics
> > > created for
> > > > > > > DLQ/Retry purposes.
> > > > > > >
> > > > > > >
> > > > > > > Thanks
> > > > > > > Yubiao Feng
> > > > > > >
> > > > >
> > >
>
Re: [DISCUSS] PIP-263: Just auto-create no-partitioned DLQ And Prevent auto-create a DLQ for a DLQ
Posted by Michael Marshall <mm...@apache.org>.
> In general, these problems hint to me that we need better definitions
> for "system" and "special" topics that break the rules of auto created
> topics in well defined ways.
This point reminds me of this discussion from a year and a half ago:
https://lists.apache.org/thread/qgbpzr6o3k5rqbs2jvpkdh8hr9jpnw39
For PIP 124, I challenged a feature based on the question of whether
the DLQ is a special topic or not.
Ultimately, we determined it is a special topic (see Matteo's response
later in that thread).
I think it is interesting because it feels like we continue to find exceptions.
Just this week, we found yet another and lamented the lack of design
for these topics:
https://github.com/apache/pulsar/pull/20514#issuecomment-1579937937.
I mention those points because I am concerned that creating a DLQ
specific piece of boolean metadata in the topic properties does not
move us toward a unified and extensible abstraction.
Unless we want to continue to infer everything from topic names, it
seems like we almost need a "topic type" enum to capture the many
types of special topics we have. Type has the same issues I mentioned
in my last email, but it could help create a more generic solution.
> ### Properties of Topic
> ```properties
> "wasAutoCreated": boolean
> ```
I don't have the answer yet, but I think we should spend more time
working on a general solution for topic classification before adding
per topic metadata.
I'd be interested to know what others think on the subject.
Thanks,
Michael
On Thu, Jun 1, 2023 at 2:08 AM Yubiao Feng
<yu...@streamnative.io.invalid> wrote:
>
> Hi Michael, Enrico
>
> > The identifier is the name. The topics have `-DLQ` and `-RETRY`
> > as suffixes, which makes them "special" (which reminds me of this
> > thread [0]). We could choose to always make these non partitioned
> > or to make them configurable in some other way.
>
> I changed the design like this:
>
> ### Properties of Topic
> ```properties
> "wasAutoCreated": boolean
> ```
>
> - If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned
> topics will be automatically created
> - If the property `wasAutoCreated` of the topic is true and the topic name
> ends with "-RETRY" or "-DLQ", retry topics and dead letter queues
> will no longer be created for this topic.
>
> How are you feeling?
>
> Thanks
> Yubiao Feng
>
> On Thu, Jun 1, 2023 at 12:48 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > We've had several bugs related to this topic of auto created
> > partitioned topics recently. These three PRs come to mind:
> >
> > * https://github.com/apache/pulsar/pull/20370
> > * https://github.com/apache/pulsar/pull/20392
> > * https://github.com/apache/pulsar/pull/20397
> >
> > Those PRs rely on server side inference based on the topic name to
> > know that a topic should not be a partitioned topic.
> >
> > In general, these problems hint to me that we need better definitions
> > for "system" and "special" topics that break the rules of auto created
> > topics in well defined ways.
> >
> > I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead.
> >
> > > I agree with you, but now the problem is that we still need
> > > an identifier to say that it is a DLQ. Do you have some
> > > suggestions?
> >
> > The identifier is the name. The topics have `-DLQ` and `-RETRY` as
> > suffixes, which makes them "special" (which reminds me of this thread
> > [0]). We could choose to always make these non partitioned or to make
> > them configurable in some other way.
> >
> > Or, we could consider giving producers and consumers the option to
> > configure a `partitionCountHint`. This could be used by the broker
> > during auto creation to create a partitioned topic with x partitions
> > when greater than 0 and to create a non-partitioned topic when 0. This
> > moves the logic out of inference on topic names and into the realm of
> > client configuration. However, it could create hard to debug
> > scenarios.
> >
> > Thanks,
> > Michael
> >
> > [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w
> >
> > On Wed, May 31, 2023 at 7:42 AM Yubiao Feng
> > <yu...@streamnative.io.invalid> wrote:
> > >
> > > Hi @Enrico
> > >
> > > > I think that it is better to add flags like:
> > > > - allowAutoTopiCreation: default "true", if "false" the broker
> > > > won't create the topic in any case
> > > > autoTopicCreationMode: undefined/partitioned/non-partitioned
> > >
> > > I agree with you, but now the problem is that we still need
> > > an identifier to say that it is a DLQ. Do you have some
> > > suggestions?
> > >
> > > Thanks
> > >
> > > Yubiao Feng
> > >
> > >
> > > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > >
> > > > Yubiao,
> > > > I understand your problem: DLQ and retry topic don't play well with
> > > > automatic topic creation.
> > > >
> > > > We are introducing wire protocol changes and also starting to add
> > > > topic properties for every new topic (they come with a cost on
> > > > metadata).
> > > >
> > > > I am not sure that adding the purposeOfAutoCreatedTopic property adds
> > > > much value and I suggest to drop it.
> > > > Logging on the broker that you created the topic is enough. I don't
> > > > think that any application or even the broker would use that piece of
> > > > information.
> > > >
> > > >
> > > > Regarding topicPurpose I think that having this "list" of
> > > > possibilities will become hard to maintain un the future.
> > > >
> > > > I think that it is better to add flags like:
> > > > - allowAutoTopiCreation: default "true", if "false" the broker won't
> > > > create the topic in any case
> > > > - autoTopicCreationMode: undefined/partitioned/non-partitioned
> > > >
> > > > One problem with partitioned topic and DLQ is that you probably want
> > > > the same number of partitions on the side topic as the original topic,
> > > > so probably you have to carry that parameter (number of partitions).
> > > >
> > > > IIRC the Subscribe command is sent directly per each partition, so I
> > > > don't know how these things may work well together.
> > > >
> > > > Enrico
> > > >
> > > >
> > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng
> > > > <xi...@apache.org> ha scritto:
> > > > >
> > > > > Hi Yubiao,
> > > > >
> > > > > Appreciate your effort in initiating this PIP. I believe these
> > changes
> > > > will
> > > > > address the existing issues and make DLQ and Retry Topic handling
> > more
> > > > > efficient and straightforward.
> > > > >
> > > > > The goals you outlined are clear and, upon implementation, will
> > improve
> > > > the
> > > > > overall functionality of Pulsar. The proposed API changes also seem
> > > > > suitable for achieving the desired outcomes.
> > > > >
> > > > > Looking forward to the progress on this PIP.
> > > > >
> > > > > Best regards,
> > > > > Xiangying
> > > > >
> > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng
> > > > > <yu...@streamnative.io.invalid> wrote:
> > > > >
> > > > > > Hi community
> > > > > >
> > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And
> > Prevent
> > > > > > auto-create a DLQ for a DLQ".
> > > > > >
> > > > > > PIP link: https://github.com/apache/pulsar/issues/20033
> > > > > >
> > > > > > ### Motivation
> > > > > >
> > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic
> > > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will
> > > > auto-create a
> > > > > > topic when the client loads it; After setting config
> > > > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`,
> > > > Pulsar
> > > > > > will auto-create a partitioned topic(which have two partitions)
> > when
> > > > the
> > > > > > client loads it.
> > > > > >
> > > > > > After the above, if using the feature [Retry Topic](
> > > > > >
> > > > > >
> > > >
> > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic
> > > > > > )
> > > > > > and [DLQ](
> > > > > >
> > > >
> > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic
> > > > > > )
> > > > > > enable topic auto-creation, we will get a partitioned DLQ and a
> > > > partitioned
> > > > > > Retry Topic like this:
> > > > > > - `{primary_topic_name}-{sub_name}-DLQ`
> > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0`
> > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1`
> > > > > > - `{primary_topic_name}-{sub_name}-RETRY`
> > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0`
> > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1`
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > I feel that almost all users will not use the multi-partitioned
> > DLQ or
> > > > > > multi-partitioned Retry topic because there is a bug that causes
> > the
> > > > above
> > > > > > behavior to be incorrect, but we have yet to receive any issues
> > about
> > > > it.
> > > > > > This bug causes the above behavior to look like this: When the
> > > > partitioned
> > > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will
> > > > create a
> > > > > > partitioned topic meta which has two partitioned but only create a
> > > > topic
> > > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic
> > named
> > > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at
> > this
> > > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed
> > bug
> > > > > > description.
> > > > > >
> > > > > > So I want to change the behavior to Just auto-create no-partitioned
> > > > > > DLQ/Retry Topic.
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > #### Prevent auto-create the DLQ for a DLQ
> > > > > > Please look at this [Discussion](
> > > > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8)
> > for
> > > > the
> > > > > > detail.
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > ### Goal
> > > > > >
> > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other
> > words:
> > > > > > prevent auto-create partitioned DLQ)
> > > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic
> > > > > > - roles:
> > > > > > - DLQ will not auto-create for a DLQ
> > > > > > - Retry Topic will not auto-create for a Retry Topic
> > > > > > - DLQ will not auto-create for a Retry Topic
> > > > > > - Retry Topic will not auto-create for a DLQ
> > > > > > - client changes: Clients will not create a DLQ for a DLQ
> > > > > > - broker changes: rejected the request which wants to
> > auto-create a
> > > > DLQ
> > > > > > for a DLQ
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > ### API Changes
> > > > > >
> > > > > > #### CommandSubscribe.java
> > > > > > ```java
> > > > > > /**
> > > > > > * This is an enumeration value with tree options: "standard",
> > "dead
> > > > > > letter", "retry letter".
> > > > > > */
> > > > > > private String topicPurpose;
> > > > > > ```
> > > > > >
> > > > > > #### Properties of Topic
> > > > > > ```properties
> > > > > > "purposeOfAutoCreatedTopic": value with tree options: "standard",
> > "dead
> > > > > > letter", "retry letter"
> > > > > > ```
> > > > > >
> > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`?
> > > > > > Because there is a scenario like this: auto-create a topic, use it
> > as
> > > > a DLQ
> > > > > > after a few days, and not use it as a DLQ after a few days, this
> > Topic
> > > > will
> > > > > > be allowed to have DLQ/Retry Topic. We only mark the topics
> > created for
> > > > > > DLQ/Retry purposes.
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > Yubiao Feng
> > > > > >
> > > >
> >
Re: [DISCUSS] PIP-263: Just auto-create no-partitioned DLQ And Prevent auto-create a DLQ for a DLQ
Posted by Yubiao Feng <yu...@streamnative.io.INVALID>.
Hi Michael, Enrico
> The identifier is the name. The topics have `-DLQ` and `-RETRY`
> as suffixes, which makes them "special" (which reminds me of this
> thread [0]). We could choose to always make these non partitioned
> or to make them configurable in some other way.
I changed the design like this:
### Properties of Topic
```properties
"wasAutoCreated": boolean
```
- If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned
topics will be automatically created
- If the property `wasAutoCreated` of the topic is true and the topic name
ends with "-RETRY" or "-DLQ", retry topics and dead letter queues
will no longer be created for this topic.
How are you feeling?
Thanks
Yubiao Feng
On Thu, Jun 1, 2023 at 12:48 PM Michael Marshall <mm...@apache.org>
wrote:
> We've had several bugs related to this topic of auto created
> partitioned topics recently. These three PRs come to mind:
>
> * https://github.com/apache/pulsar/pull/20370
> * https://github.com/apache/pulsar/pull/20392
> * https://github.com/apache/pulsar/pull/20397
>
> Those PRs rely on server side inference based on the topic name to
> know that a topic should not be a partitioned topic.
>
> In general, these problems hint to me that we need better definitions
> for "system" and "special" topics that break the rules of auto created
> topics in well defined ways.
>
> I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead.
>
> > I agree with you, but now the problem is that we still need
> > an identifier to say that it is a DLQ. Do you have some
> > suggestions?
>
> The identifier is the name. The topics have `-DLQ` and `-RETRY` as
> suffixes, which makes them "special" (which reminds me of this thread
> [0]). We could choose to always make these non partitioned or to make
> them configurable in some other way.
>
> Or, we could consider giving producers and consumers the option to
> configure a `partitionCountHint`. This could be used by the broker
> during auto creation to create a partitioned topic with x partitions
> when greater than 0 and to create a non-partitioned topic when 0. This
> moves the logic out of inference on topic names and into the realm of
> client configuration. However, it could create hard to debug
> scenarios.
>
> Thanks,
> Michael
>
> [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w
>
> On Wed, May 31, 2023 at 7:42 AM Yubiao Feng
> <yu...@streamnative.io.invalid> wrote:
> >
> > Hi @Enrico
> >
> > > I think that it is better to add flags like:
> > > - allowAutoTopiCreation: default "true", if "false" the broker
> > > won't create the topic in any case
> > > autoTopicCreationMode: undefined/partitioned/non-partitioned
> >
> > I agree with you, but now the problem is that we still need
> > an identifier to say that it is a DLQ. Do you have some
> > suggestions?
> >
> > Thanks
> >
> > Yubiao Feng
> >
> >
> > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eo...@gmail.com>
> wrote:
> >
> > > Yubiao,
> > > I understand your problem: DLQ and retry topic don't play well with
> > > automatic topic creation.
> > >
> > > We are introducing wire protocol changes and also starting to add
> > > topic properties for every new topic (they come with a cost on
> > > metadata).
> > >
> > > I am not sure that adding the purposeOfAutoCreatedTopic property adds
> > > much value and I suggest to drop it.
> > > Logging on the broker that you created the topic is enough. I don't
> > > think that any application or even the broker would use that piece of
> > > information.
> > >
> > >
> > > Regarding topicPurpose I think that having this "list" of
> > > possibilities will become hard to maintain un the future.
> > >
> > > I think that it is better to add flags like:
> > > - allowAutoTopiCreation: default "true", if "false" the broker won't
> > > create the topic in any case
> > > - autoTopicCreationMode: undefined/partitioned/non-partitioned
> > >
> > > One problem with partitioned topic and DLQ is that you probably want
> > > the same number of partitions on the side topic as the original topic,
> > > so probably you have to carry that parameter (number of partitions).
> > >
> > > IIRC the Subscribe command is sent directly per each partition, so I
> > > don't know how these things may work well together.
> > >
> > > Enrico
> > >
> > >
> > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng
> > > <xi...@apache.org> ha scritto:
> > > >
> > > > Hi Yubiao,
> > > >
> > > > Appreciate your effort in initiating this PIP. I believe these
> changes
> > > will
> > > > address the existing issues and make DLQ and Retry Topic handling
> more
> > > > efficient and straightforward.
> > > >
> > > > The goals you outlined are clear and, upon implementation, will
> improve
> > > the
> > > > overall functionality of Pulsar. The proposed API changes also seem
> > > > suitable for achieving the desired outcomes.
> > > >
> > > > Looking forward to the progress on this PIP.
> > > >
> > > > Best regards,
> > > > Xiangying
> > > >
> > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng
> > > > <yu...@streamnative.io.invalid> wrote:
> > > >
> > > > > Hi community
> > > > >
> > > > > I started a PIP about "Just auto-create no-partitioned DLQ And
> Prevent
> > > > > auto-create a DLQ for a DLQ".
> > > > >
> > > > > PIP link: https://github.com/apache/pulsar/issues/20033
> > > > >
> > > > > ### Motivation
> > > > >
> > > > > #### Just auto-create no-partitioned DLQ/Retry Topic
> > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will
> > > auto-create a
> > > > > topic when the client loads it; After setting config
> > > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`,
> > > Pulsar
> > > > > will auto-create a partitioned topic(which have two partitions)
> when
> > > the
> > > > > client loads it.
> > > > >
> > > > > After the above, if using the feature [Retry Topic](
> > > > >
> > > > >
> > >
> https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic
> > > > > )
> > > > > and [DLQ](
> > > > >
> > >
> https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic
> > > > > )
> > > > > enable topic auto-creation, we will get a partitioned DLQ and a
> > > partitioned
> > > > > Retry Topic like this:
> > > > > - `{primary_topic_name}-{sub_name}-DLQ`
> > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0`
> > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1`
> > > > > - `{primary_topic_name}-{sub_name}-RETRY`
> > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0`
> > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1`
> > > > >
> > > > > ----
> > > > >
> > > > > I feel that almost all users will not use the multi-partitioned
> DLQ or
> > > > > multi-partitioned Retry topic because there is a bug that causes
> the
> > > above
> > > > > behavior to be incorrect, but we have yet to receive any issues
> about
> > > it.
> > > > > This bug causes the above behavior to look like this: When the
> > > partitioned
> > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will
> > > create a
> > > > > partitioned topic meta which has two partitioned but only create a
> > > topic
> > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic
> named
> > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at
> this
> > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed
> bug
> > > > > description.
> > > > >
> > > > > So I want to change the behavior to Just auto-create no-partitioned
> > > > > DLQ/Retry Topic.
> > > > >
> > > > > ----
> > > > >
> > > > > #### Prevent auto-create the DLQ for a DLQ
> > > > > Please look at this [Discussion](
> > > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8)
> for
> > > the
> > > > > detail.
> > > > >
> > > > > ----
> > > > >
> > > > > ### Goal
> > > > >
> > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other
> words:
> > > > > prevent auto-create partitioned DLQ)
> > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic
> > > > > - roles:
> > > > > - DLQ will not auto-create for a DLQ
> > > > > - Retry Topic will not auto-create for a Retry Topic
> > > > > - DLQ will not auto-create for a Retry Topic
> > > > > - Retry Topic will not auto-create for a DLQ
> > > > > - client changes: Clients will not create a DLQ for a DLQ
> > > > > - broker changes: rejected the request which wants to
> auto-create a
> > > DLQ
> > > > > for a DLQ
> > > > >
> > > > > ----
> > > > >
> > > > > ### API Changes
> > > > >
> > > > > #### CommandSubscribe.java
> > > > > ```java
> > > > > /**
> > > > > * This is an enumeration value with tree options: "standard",
> "dead
> > > > > letter", "retry letter".
> > > > > */
> > > > > private String topicPurpose;
> > > > > ```
> > > > >
> > > > > #### Properties of Topic
> > > > > ```properties
> > > > > "purposeOfAutoCreatedTopic": value with tree options: "standard",
> "dead
> > > > > letter", "retry letter"
> > > > > ```
> > > > >
> > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`?
> > > > > Because there is a scenario like this: auto-create a topic, use it
> as
> > > a DLQ
> > > > > after a few days, and not use it as a DLQ after a few days, this
> Topic
> > > will
> > > > > be allowed to have DLQ/Retry Topic. We only mark the topics
> created for
> > > > > DLQ/Retry purposes.
> > > > >
> > > > >
> > > > > Thanks
> > > > > Yubiao Feng
> > > > >
> > >
>