You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Zike Yang <zk...@streamnative.io.INVALID> on 2022/01/25 09:34:21 UTC
[VOTE] PIP-124: Create init subscription before sending message to DLQ
Hi Pulsar Community,
This is the voting thread for PIP-124. It will stay open for at least
48 hours. Pasted below for quoting convenience.
Here is the issue for this PIP: https://github.com/apache/pulsar/issues/13408
----
## Motivation
If we enable the DLQ when consuming messages. For some messages that
can't be processed successfully, the messages will be moved to the
DLQ, but if we do not specify the data retention for the namespace or
create a subscription for the DLQ to retain the data, the data of the
DLQ will be removed automatically. Therefore, we need to create the
initial subscription before sending messages to the DLQ.
## Goal
Users can set the initial subscription name in the DeadLetterPolicy.
The consumer will create the initial subscription before sending
messages to the DLQ. At this point, subsequent messages produced to
the DLQ are not automatically deleted unexpectedly. If
`allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
subscription won't be created automatically. Otherwise, it will
confuse the user. Users can explicitly create that subscription to
handle this case.
This PIP needs to be compatible with the previous behavior. The
initial subscription name in the DeadLetterPolicy is optional. The
default behavior is not to create the initial subscription which is
consistent with the original behavior.
## API Changes
* Add `initSubscriptionName` to the `DeadLetterPolicy`
```java
/**
* Name of the initial subscription name of the dead letter topic.
* If this field is not set, the initial subscription will not be created.
*/
private String initSubscriptionName;
```
* Add a new field to the `CommandProducer`. The broker will use this
field to create the initial subscription.
```proto
optional string initial_subscription_name
```
* Add a new config to the Producer Configuration. This allows the
consumer to specify the initial subscription when creating the
deadLetterProducer.
```java
/**
* Use this config to automatically create an initial subscription
when creating the topic.
* If this field is not set, the initial subscription will not be created.
*
* @param initialSubscriptionName
* Name of the initial subscription of the topic.
* @return the producer builder instance
*/
ProducerBuilder<T> initialSubscriptionName(String initialSubscriptionName);
```
## Implementation
When the deadLetterProducer is initialized, the consumer will set the
initial subscription of the deadLetterProducer according to the
DeadLetterPolicy.
When the broker creates a producer(after receiving the
CommandProducer), if it finds that the topic does not exist, it will
not only create the topic(if the topic automatically creation is
enabled) but also create the initial subscription(if the initial
subscription name is set). When creating an initial subscription, the
user needs to have the `canConsume` permission and the
`allowAutoSubscriptionCreation` needs to be enabled.
## Reject Alternatives
### Create the initial subscription using the consumer
Before the deadLetterProducer is initialized, the consumer first tries
to create a deadLetterConsumer using the initial subscription name in
the DeadLetterPolicy. When this subscription already exists, the
ConsumerBusy exception will occur. In this case, we can ignore that
exception and create the deadLetterProducer.
This is the original solution for this PIP. But this introduces too
much workload for the client. In the subsequent discussion, we try to
add a new command `CommandCreateSubscription` to avoid creating and
closing the deadLetterConsumer, but again, this introduces workload as
well as greater complexity.
### Use the retention policy to retent the data of the DLQ
Before creating the deadLetterProducer, an admin can create a
retention policy for the topic or namespace. Then, consumers of the
topic have the duration of the retention policy to discover the topic
and create a subscription before messages are lost.
But users are not easy to set a different data retention policy or
create a new subscription for a lazy created DLQ topic.
### Add a new policy to retain data when the topic has no subscriptions
Adding a new policy for pulsar topics: a namespace or a topic level
policy that makes it possible to retain messages indefinitely when a
topic has no subscriptions.
For the auto-created topic, the subscription can only be determined at
the time of creation. It may or may not create. If users are able to
determine which consumers are, and these consumers need to receive any
message sent by the producer, they should create the topic and
subscription manually or use the consumer to trigger the topic
auto-creation, not the producer.
### Add the initial subscription name field to the metadata of the
CommandProducer
Use producer metadata to carry the init subscription name so that we
don't need to introduce a new field in CommandProducer, and the new
field looks a little confusing.
While this does not change protobuf, it still implicitly changes the
protocol. This solution has different advantages and disadvantages
than the current one. Finally, we decided to go with the current
solution.
In addition, there is a discussion about whether we need to create the
initial subscription on the retryLetterTopic. The consumer has already
subscribed to the retryLetterTopic and created the subscription before
creating the retryLetterProducer, so we don't need to create the
initial subscription for it.
Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
----
Thanks
Zike Yang
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Zike Yang <zk...@streamnative.io.INVALID>.
Hi all,
Since there are no objections on this PIP and this vote has been
staying open for 48 hours.
Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
Thanks for your participation.
Thanks,
Zike
On Thu, Jan 27, 2022 at 10:09 AM Zike Yang <zk...@streamnative.io> wrote:
>
> Hi Michael,
>
> Sorry, I forgot about it. Sorry for my mistake.
> Thanks for your correction. I will stay this vote open.
>
> Thanks,
> Zike
>
>
>
>
>
> On Thu, Jan 27, 2022 at 12:31 AM Michael Marshall <mm...@apache.org> wrote:
> >
> > Hi Zike,
> >
> > Based on our current voting rules, I believe we need to leave the vote
> > open for at least 17 more hours.
> >
> > There has been discussion on the dev list about changing this rule,
> > but I don't believe we have reached any conclusions yet.
> >
> > Thanks,
> > Michael
> >
> > On Wed, Jan 26, 2022 at 6:04 AM Zike Yang
> > <zk...@streamnative.io.invalid> wrote:
> > >
> > > Thanks for your participation and your great efforts to move this forward!
> > > Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
> > >
> > > Thanks,
> > > Zike
> > >
> > > On Wed, Jan 26, 2022 at 3:54 PM PengHui Li <pe...@apache.org> wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > > On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mm...@apache.org>
> > > > wrote:
> > > >
> > > > > +1 (non binding) - this proposal looks great! Thank you for a good
> > > > > discussion of this feature!
> > > > >
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Hang
> > > > > >
> > > > > > Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding)
> > > > > > > >
> > > > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zkyang@streamnative.io
> > > > > .INVALID>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Pulsar Community,
> > > > > > > > >
> > > > > > > > > This is the voting thread for PIP-124. It will stay open for at
> > > > > least
> > > > > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > > > > >
> > > > > > > > > Here is the issue for this PIP:
> > > > > > > > https://github.com/apache/pulsar/issues/13408
> > > > > > > > >
> > > > > > > > > ----
> > > > > > > > >
> > > > > > > > > ## Motivation
> > > > > > > > >
> > > > > > > > > If we enable the DLQ when consuming messages. For some messages
> > > > > that
> > > > > > > > > can't be processed successfully, the messages will be moved to the
> > > > > > > > > DLQ, but if we do not specify the data retention for the namespace
> > > > > or
> > > > > > > > > create a subscription for the DLQ to retain the data, the data of
> > > > > the
> > > > > > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > > > > > initial subscription before sending messages to the DLQ.
> > > > > > > > >
> > > > > > > > > ## Goal
> > > > > > > > >
> > > > > > > > > Users can set the initial subscription name in the
> > > > > DeadLetterPolicy.
> > > > > > > > > The consumer will create the initial subscription before sending
> > > > > > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > > > > initial
> > > > > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > > > > confuse the user. Users can explicitly create that subscription to
> > > > > > > > > handle this case.
> > > > > > > > >
> > > > > > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > > > > > default behavior is not to create the initial subscription which is
> > > > > > > > > consistent with the original behavior.
> > > > > > > > >
> > > > > > > > > ## API Changes
> > > > > > > > >
> > > > > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > > > > > >
> > > > > > > > > ```java
> > > > > > > > > /**
> > > > > > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > > > > > * If this field is not set, the initial subscription will not be
> > > > > created.
> > > > > > > > > */
> > > > > > > > > private String initSubscriptionName;
> > > > > > > > > ```
> > > > > > > > >
> > > > > > > > > * Add a new field to the `CommandProducer`. The broker will use
> > > > > this
> > > > > > > > > field to create the initial subscription.
> > > > > > > > >
> > > > > > > > > ```proto
> > > > > > > > > optional string initial_subscription_name
> > > > > > > > > ```
> > > > > > > > >
> > > > > > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > > > > > consumer to specify the initial subscription when creating the
> > > > > > > > > deadLetterProducer.
> > > > > > > > >
> > > > > > > > > ```java
> > > > > > > > > /**
> > > > > > > > > * Use this config to automatically create an initial subscription
> > > > > > > > > when creating the topic.
> > > > > > > > > * If this field is not set, the initial subscription will not be
> > > > > created.
> > > > > > > > > *
> > > > > > > > > * @param initialSubscriptionName
> > > > > > > > > * Name of the initial subscription of the topic.
> > > > > > > > > * @return the producer builder instance
> > > > > > > > > */
> > > > > > > > > ProducerBuilder<T> initialSubscriptionName(String
> > > > > > > > initialSubscriptionName);
> > > > > > > > > ```
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > ## Implementation
> > > > > > > > >
> > > > > > > > > When the deadLetterProducer is initialized, the consumer will set
> > > > > the
> > > > > > > > > initial subscription of the deadLetterProducer according to the
> > > > > > > > > DeadLetterPolicy.
> > > > > > > > >
> > > > > > > > > When the broker creates a producer(after receiving the
> > > > > > > > > CommandProducer), if it finds that the topic does not exist, it
> > > > > will
> > > > > > > > > not only create the topic(if the topic automatically creation is
> > > > > > > > > enabled) but also create the initial subscription(if the initial
> > > > > > > > > subscription name is set). When creating an initial subscription,
> > > > > the
> > > > > > > > > user needs to have the `canConsume` permission and the
> > > > > > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > > > > > >
> > > > > > > > > ## Reject Alternatives
> > > > > > > > >
> > > > > > > > > ### Create the initial subscription using the consumer
> > > > > > > > >
> > > > > > > > > Before the deadLetterProducer is initialized, the consumer first
> > > > > tries
> > > > > > > > > to create a deadLetterConsumer using the initial subscription name
> > > > > in
> > > > > > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > > > > > exception and create the deadLetterProducer.
> > > > > > > > >
> > > > > > > > > This is the original solution for this PIP. But this introduces too
> > > > > > > > > much workload for the client. In the subsequent discussion, we try
> > > > > to
> > > > > > > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > > > > > > closing the deadLetterConsumer, but again, this introduces
> > > > > workload as
> > > > > > > > > well as greater complexity.
> > > > > > > > >
> > > > > > > > > ### Use the retention policy to retent the data of the DLQ
> > > > > > > > >
> > > > > > > > > Before creating the deadLetterProducer, an admin can create a
> > > > > > > > > retention policy for the topic or namespace. Then, consumers of the
> > > > > > > > > topic have the duration of the retention policy to discover the
> > > > > topic
> > > > > > > > > and create a subscription before messages are lost.
> > > > > > > > >
> > > > > > > > > But users are not easy to set a different data retention policy or
> > > > > > > > > create a new subscription for a lazy created DLQ topic.
> > > > > > > > >
> > > > > > > > > ### Add a new policy to retain data when the topic has no
> > > > > subscriptions
> > > > > > > > >
> > > > > > > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > > > > > > policy that makes it possible to retain messages indefinitely when
> > > > > a
> > > > > > > > > topic has no subscriptions.
> > > > > > > > >
> > > > > > > > > For the auto-created topic, the subscription can only be
> > > > > determined at
> > > > > > > > > the time of creation. It may or may not create. If users are able
> > > > > to
> > > > > > > > > determine which consumers are, and these consumers need to receive
> > > > > any
> > > > > > > > > message sent by the producer, they should create the topic and
> > > > > > > > > subscription manually or use the consumer to trigger the topic
> > > > > > > > > auto-creation, not the producer.
> > > > > > > > >
> > > > > > > > > ### Add the initial subscription name field to the metadata of the
> > > > > > > > > CommandProducer
> > > > > > > > >
> > > > > > > > > Use producer metadata to carry the init subscription name so that
> > > > > we
> > > > > > > > > don't need to introduce a new field in CommandProducer, and the new
> > > > > > > > > field looks a little confusing.
> > > > > > > > >
> > > > > > > > > While this does not change protobuf, it still implicitly changes
> > > > > the
> > > > > > > > > protocol. This solution has different advantages and disadvantages
> > > > > > > > > than the current one. Finally, we decided to go with the current
> > > > > > > > > solution.
> > > > > > > > >
> > > > > > > > > In addition, there is a discussion about whether we need to create
> > > > > the
> > > > > > > > > initial subscription on the retryLetterTopic. The consumer has
> > > > > already
> > > > > > > > > subscribed to the retryLetterTopic and created the subscription
> > > > > before
> > > > > > > > > creating the retryLetterProducer, so we don't need to create the
> > > > > > > > > initial subscription for it.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Prototype implementation PR:
> > > > > https://github.com/apache/pulsar/pull/13355
> > > > > > > > >
> > > > > > > > > ----
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Zike Yang
> > > > > > > >
> > > > > > > >
> > > > >
> > >
> > >
> > >
> > > --
> > > Zike Yang
>
>
>
> --
> Zike Yang
--
Zike Yang
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Zike Yang <zk...@streamnative.io.INVALID>.
Hi Michael,
Sorry, I forgot about it. Sorry for my mistake.
Thanks for your correction. I will stay this vote open.
Thanks,
Zike
On Thu, Jan 27, 2022 at 12:31 AM Michael Marshall <mm...@apache.org> wrote:
>
> Hi Zike,
>
> Based on our current voting rules, I believe we need to leave the vote
> open for at least 17 more hours.
>
> There has been discussion on the dev list about changing this rule,
> but I don't believe we have reached any conclusions yet.
>
> Thanks,
> Michael
>
> On Wed, Jan 26, 2022 at 6:04 AM Zike Yang
> <zk...@streamnative.io.invalid> wrote:
> >
> > Thanks for your participation and your great efforts to move this forward!
> > Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
> >
> > Thanks,
> > Zike
> >
> > On Wed, Jan 26, 2022 at 3:54 PM PengHui Li <pe...@apache.org> wrote:
> > >
> > > +1 (binding)
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mm...@apache.org>
> > > wrote:
> > >
> > > > +1 (non binding) - this proposal looks great! Thank you for a good
> > > > discussion of this feature!
> > > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks,
> > > > > Hang
> > > > >
> > > > > Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zkyang@streamnative.io
> > > > .INVALID>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Pulsar Community,
> > > > > > > >
> > > > > > > > This is the voting thread for PIP-124. It will stay open for at
> > > > least
> > > > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > > > >
> > > > > > > > Here is the issue for this PIP:
> > > > > > > https://github.com/apache/pulsar/issues/13408
> > > > > > > >
> > > > > > > > ----
> > > > > > > >
> > > > > > > > ## Motivation
> > > > > > > >
> > > > > > > > If we enable the DLQ when consuming messages. For some messages
> > > > that
> > > > > > > > can't be processed successfully, the messages will be moved to the
> > > > > > > > DLQ, but if we do not specify the data retention for the namespace
> > > > or
> > > > > > > > create a subscription for the DLQ to retain the data, the data of
> > > > the
> > > > > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > > > > initial subscription before sending messages to the DLQ.
> > > > > > > >
> > > > > > > > ## Goal
> > > > > > > >
> > > > > > > > Users can set the initial subscription name in the
> > > > DeadLetterPolicy.
> > > > > > > > The consumer will create the initial subscription before sending
> > > > > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > > > initial
> > > > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > > > confuse the user. Users can explicitly create that subscription to
> > > > > > > > handle this case.
> > > > > > > >
> > > > > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > > > > default behavior is not to create the initial subscription which is
> > > > > > > > consistent with the original behavior.
> > > > > > > >
> > > > > > > > ## API Changes
> > > > > > > >
> > > > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > > > > >
> > > > > > > > ```java
> > > > > > > > /**
> > > > > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > > > > * If this field is not set, the initial subscription will not be
> > > > created.
> > > > > > > > */
> > > > > > > > private String initSubscriptionName;
> > > > > > > > ```
> > > > > > > >
> > > > > > > > * Add a new field to the `CommandProducer`. The broker will use
> > > > this
> > > > > > > > field to create the initial subscription.
> > > > > > > >
> > > > > > > > ```proto
> > > > > > > > optional string initial_subscription_name
> > > > > > > > ```
> > > > > > > >
> > > > > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > > > > consumer to specify the initial subscription when creating the
> > > > > > > > deadLetterProducer.
> > > > > > > >
> > > > > > > > ```java
> > > > > > > > /**
> > > > > > > > * Use this config to automatically create an initial subscription
> > > > > > > > when creating the topic.
> > > > > > > > * If this field is not set, the initial subscription will not be
> > > > created.
> > > > > > > > *
> > > > > > > > * @param initialSubscriptionName
> > > > > > > > * Name of the initial subscription of the topic.
> > > > > > > > * @return the producer builder instance
> > > > > > > > */
> > > > > > > > ProducerBuilder<T> initialSubscriptionName(String
> > > > > > > initialSubscriptionName);
> > > > > > > > ```
> > > > > > > >
> > > > > > > >
> > > > > > > > ## Implementation
> > > > > > > >
> > > > > > > > When the deadLetterProducer is initialized, the consumer will set
> > > > the
> > > > > > > > initial subscription of the deadLetterProducer according to the
> > > > > > > > DeadLetterPolicy.
> > > > > > > >
> > > > > > > > When the broker creates a producer(after receiving the
> > > > > > > > CommandProducer), if it finds that the topic does not exist, it
> > > > will
> > > > > > > > not only create the topic(if the topic automatically creation is
> > > > > > > > enabled) but also create the initial subscription(if the initial
> > > > > > > > subscription name is set). When creating an initial subscription,
> > > > the
> > > > > > > > user needs to have the `canConsume` permission and the
> > > > > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > > > > >
> > > > > > > > ## Reject Alternatives
> > > > > > > >
> > > > > > > > ### Create the initial subscription using the consumer
> > > > > > > >
> > > > > > > > Before the deadLetterProducer is initialized, the consumer first
> > > > tries
> > > > > > > > to create a deadLetterConsumer using the initial subscription name
> > > > in
> > > > > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > > > > exception and create the deadLetterProducer.
> > > > > > > >
> > > > > > > > This is the original solution for this PIP. But this introduces too
> > > > > > > > much workload for the client. In the subsequent discussion, we try
> > > > to
> > > > > > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > > > > > closing the deadLetterConsumer, but again, this introduces
> > > > workload as
> > > > > > > > well as greater complexity.
> > > > > > > >
> > > > > > > > ### Use the retention policy to retent the data of the DLQ
> > > > > > > >
> > > > > > > > Before creating the deadLetterProducer, an admin can create a
> > > > > > > > retention policy for the topic or namespace. Then, consumers of the
> > > > > > > > topic have the duration of the retention policy to discover the
> > > > topic
> > > > > > > > and create a subscription before messages are lost.
> > > > > > > >
> > > > > > > > But users are not easy to set a different data retention policy or
> > > > > > > > create a new subscription for a lazy created DLQ topic.
> > > > > > > >
> > > > > > > > ### Add a new policy to retain data when the topic has no
> > > > subscriptions
> > > > > > > >
> > > > > > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > > > > > policy that makes it possible to retain messages indefinitely when
> > > > a
> > > > > > > > topic has no subscriptions.
> > > > > > > >
> > > > > > > > For the auto-created topic, the subscription can only be
> > > > determined at
> > > > > > > > the time of creation. It may or may not create. If users are able
> > > > to
> > > > > > > > determine which consumers are, and these consumers need to receive
> > > > any
> > > > > > > > message sent by the producer, they should create the topic and
> > > > > > > > subscription manually or use the consumer to trigger the topic
> > > > > > > > auto-creation, not the producer.
> > > > > > > >
> > > > > > > > ### Add the initial subscription name field to the metadata of the
> > > > > > > > CommandProducer
> > > > > > > >
> > > > > > > > Use producer metadata to carry the init subscription name so that
> > > > we
> > > > > > > > don't need to introduce a new field in CommandProducer, and the new
> > > > > > > > field looks a little confusing.
> > > > > > > >
> > > > > > > > While this does not change protobuf, it still implicitly changes
> > > > the
> > > > > > > > protocol. This solution has different advantages and disadvantages
> > > > > > > > than the current one. Finally, we decided to go with the current
> > > > > > > > solution.
> > > > > > > >
> > > > > > > > In addition, there is a discussion about whether we need to create
> > > > the
> > > > > > > > initial subscription on the retryLetterTopic. The consumer has
> > > > already
> > > > > > > > subscribed to the retryLetterTopic and created the subscription
> > > > before
> > > > > > > > creating the retryLetterProducer, so we don't need to create the
> > > > > > > > initial subscription for it.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Prototype implementation PR:
> > > > https://github.com/apache/pulsar/pull/13355
> > > > > > > >
> > > > > > > > ----
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Zike Yang
> > > > > > >
> > > > > > >
> > > >
> >
> >
> >
> > --
> > Zike Yang
--
Zike Yang
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Michael Marshall <mm...@apache.org>.
Hi Zike,
Based on our current voting rules, I believe we need to leave the vote
open for at least 17 more hours.
There has been discussion on the dev list about changing this rule,
but I don't believe we have reached any conclusions yet.
Thanks,
Michael
On Wed, Jan 26, 2022 at 6:04 AM Zike Yang
<zk...@streamnative.io.invalid> wrote:
>
> Thanks for your participation and your great efforts to move this forward!
> Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
>
> Thanks,
> Zike
>
> On Wed, Jan 26, 2022 at 3:54 PM PengHui Li <pe...@apache.org> wrote:
> >
> > +1 (binding)
> >
> > Regards,
> > Penghui
> >
> > On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mm...@apache.org>
> > wrote:
> >
> > > +1 (non binding) - this proposal looks great! Thank you for a good
> > > discussion of this feature!
> > >
> > > Thanks,
> > > Michael
> > >
> > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
> > > >
> > > > +1 (binding)
> > > >
> > > > Thanks,
> > > > Hang
> > > >
> > > > Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> > > > >
> > > > > +1
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zkyang@streamnative.io
> > > .INVALID>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Pulsar Community,
> > > > > > >
> > > > > > > This is the voting thread for PIP-124. It will stay open for at
> > > least
> > > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > > >
> > > > > > > Here is the issue for this PIP:
> > > > > > https://github.com/apache/pulsar/issues/13408
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > ## Motivation
> > > > > > >
> > > > > > > If we enable the DLQ when consuming messages. For some messages
> > > that
> > > > > > > can't be processed successfully, the messages will be moved to the
> > > > > > > DLQ, but if we do not specify the data retention for the namespace
> > > or
> > > > > > > create a subscription for the DLQ to retain the data, the data of
> > > the
> > > > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > > > initial subscription before sending messages to the DLQ.
> > > > > > >
> > > > > > > ## Goal
> > > > > > >
> > > > > > > Users can set the initial subscription name in the
> > > DeadLetterPolicy.
> > > > > > > The consumer will create the initial subscription before sending
> > > > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > > initial
> > > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > > confuse the user. Users can explicitly create that subscription to
> > > > > > > handle this case.
> > > > > > >
> > > > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > > > default behavior is not to create the initial subscription which is
> > > > > > > consistent with the original behavior.
> > > > > > >
> > > > > > > ## API Changes
> > > > > > >
> > > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > > > >
> > > > > > > ```java
> > > > > > > /**
> > > > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > > > * If this field is not set, the initial subscription will not be
> > > created.
> > > > > > > */
> > > > > > > private String initSubscriptionName;
> > > > > > > ```
> > > > > > >
> > > > > > > * Add a new field to the `CommandProducer`. The broker will use
> > > this
> > > > > > > field to create the initial subscription.
> > > > > > >
> > > > > > > ```proto
> > > > > > > optional string initial_subscription_name
> > > > > > > ```
> > > > > > >
> > > > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > > > consumer to specify the initial subscription when creating the
> > > > > > > deadLetterProducer.
> > > > > > >
> > > > > > > ```java
> > > > > > > /**
> > > > > > > * Use this config to automatically create an initial subscription
> > > > > > > when creating the topic.
> > > > > > > * If this field is not set, the initial subscription will not be
> > > created.
> > > > > > > *
> > > > > > > * @param initialSubscriptionName
> > > > > > > * Name of the initial subscription of the topic.
> > > > > > > * @return the producer builder instance
> > > > > > > */
> > > > > > > ProducerBuilder<T> initialSubscriptionName(String
> > > > > > initialSubscriptionName);
> > > > > > > ```
> > > > > > >
> > > > > > >
> > > > > > > ## Implementation
> > > > > > >
> > > > > > > When the deadLetterProducer is initialized, the consumer will set
> > > the
> > > > > > > initial subscription of the deadLetterProducer according to the
> > > > > > > DeadLetterPolicy.
> > > > > > >
> > > > > > > When the broker creates a producer(after receiving the
> > > > > > > CommandProducer), if it finds that the topic does not exist, it
> > > will
> > > > > > > not only create the topic(if the topic automatically creation is
> > > > > > > enabled) but also create the initial subscription(if the initial
> > > > > > > subscription name is set). When creating an initial subscription,
> > > the
> > > > > > > user needs to have the `canConsume` permission and the
> > > > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > > > >
> > > > > > > ## Reject Alternatives
> > > > > > >
> > > > > > > ### Create the initial subscription using the consumer
> > > > > > >
> > > > > > > Before the deadLetterProducer is initialized, the consumer first
> > > tries
> > > > > > > to create a deadLetterConsumer using the initial subscription name
> > > in
> > > > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > > > exception and create the deadLetterProducer.
> > > > > > >
> > > > > > > This is the original solution for this PIP. But this introduces too
> > > > > > > much workload for the client. In the subsequent discussion, we try
> > > to
> > > > > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > > > > closing the deadLetterConsumer, but again, this introduces
> > > workload as
> > > > > > > well as greater complexity.
> > > > > > >
> > > > > > > ### Use the retention policy to retent the data of the DLQ
> > > > > > >
> > > > > > > Before creating the deadLetterProducer, an admin can create a
> > > > > > > retention policy for the topic or namespace. Then, consumers of the
> > > > > > > topic have the duration of the retention policy to discover the
> > > topic
> > > > > > > and create a subscription before messages are lost.
> > > > > > >
> > > > > > > But users are not easy to set a different data retention policy or
> > > > > > > create a new subscription for a lazy created DLQ topic.
> > > > > > >
> > > > > > > ### Add a new policy to retain data when the topic has no
> > > subscriptions
> > > > > > >
> > > > > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > > > > policy that makes it possible to retain messages indefinitely when
> > > a
> > > > > > > topic has no subscriptions.
> > > > > > >
> > > > > > > For the auto-created topic, the subscription can only be
> > > determined at
> > > > > > > the time of creation. It may or may not create. If users are able
> > > to
> > > > > > > determine which consumers are, and these consumers need to receive
> > > any
> > > > > > > message sent by the producer, they should create the topic and
> > > > > > > subscription manually or use the consumer to trigger the topic
> > > > > > > auto-creation, not the producer.
> > > > > > >
> > > > > > > ### Add the initial subscription name field to the metadata of the
> > > > > > > CommandProducer
> > > > > > >
> > > > > > > Use producer metadata to carry the init subscription name so that
> > > we
> > > > > > > don't need to introduce a new field in CommandProducer, and the new
> > > > > > > field looks a little confusing.
> > > > > > >
> > > > > > > While this does not change protobuf, it still implicitly changes
> > > the
> > > > > > > protocol. This solution has different advantages and disadvantages
> > > > > > > than the current one. Finally, we decided to go with the current
> > > > > > > solution.
> > > > > > >
> > > > > > > In addition, there is a discussion about whether we need to create
> > > the
> > > > > > > initial subscription on the retryLetterTopic. The consumer has
> > > already
> > > > > > > subscribed to the retryLetterTopic and created the subscription
> > > before
> > > > > > > creating the retryLetterProducer, so we don't need to create the
> > > > > > > initial subscription for it.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Prototype implementation PR:
> > > https://github.com/apache/pulsar/pull/13355
> > > > > > >
> > > > > > > ----
> > > > > > >
> > > > > > > Thanks
> > > > > > > Zike Yang
> > > > > >
> > > > > >
> > >
>
>
>
> --
> Zike Yang
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Zike Yang <zk...@streamnative.io.INVALID>.
Thanks for your participation and your great efforts to move this forward!
Close the vote with 3 (+1) bindings and 2 (+1) non-bindings.
Thanks,
Zike
On Wed, Jan 26, 2022 at 3:54 PM PengHui Li <pe...@apache.org> wrote:
>
> +1 (binding)
>
> Regards,
> Penghui
>
> On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > +1 (non binding) - this proposal looks great! Thank you for a good
> > discussion of this feature!
> >
> > Thanks,
> > Michael
> >
> > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Hang
> > >
> > > Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> > > >
> > > > +1
> > > >
> > > >
> > > >
> > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zkyang@streamnative.io
> > .INVALID>
> > > > > wrote:
> > > > > >
> > > > > > Hi Pulsar Community,
> > > > > >
> > > > > > This is the voting thread for PIP-124. It will stay open for at
> > least
> > > > > > 48 hours. Pasted below for quoting convenience.
> > > > > >
> > > > > > Here is the issue for this PIP:
> > > > > https://github.com/apache/pulsar/issues/13408
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > ## Motivation
> > > > > >
> > > > > > If we enable the DLQ when consuming messages. For some messages
> > that
> > > > > > can't be processed successfully, the messages will be moved to the
> > > > > > DLQ, but if we do not specify the data retention for the namespace
> > or
> > > > > > create a subscription for the DLQ to retain the data, the data of
> > the
> > > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > > initial subscription before sending messages to the DLQ.
> > > > > >
> > > > > > ## Goal
> > > > > >
> > > > > > Users can set the initial subscription name in the
> > DeadLetterPolicy.
> > > > > > The consumer will create the initial subscription before sending
> > > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> > initial
> > > > > > subscription won't be created automatically. Otherwise, it will
> > > > > > confuse the user. Users can explicitly create that subscription to
> > > > > > handle this case.
> > > > > >
> > > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > > default behavior is not to create the initial subscription which is
> > > > > > consistent with the original behavior.
> > > > > >
> > > > > > ## API Changes
> > > > > >
> > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > > >
> > > > > > ```java
> > > > > > /**
> > > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > > * If this field is not set, the initial subscription will not be
> > created.
> > > > > > */
> > > > > > private String initSubscriptionName;
> > > > > > ```
> > > > > >
> > > > > > * Add a new field to the `CommandProducer`. The broker will use
> > this
> > > > > > field to create the initial subscription.
> > > > > >
> > > > > > ```proto
> > > > > > optional string initial_subscription_name
> > > > > > ```
> > > > > >
> > > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > > consumer to specify the initial subscription when creating the
> > > > > > deadLetterProducer.
> > > > > >
> > > > > > ```java
> > > > > > /**
> > > > > > * Use this config to automatically create an initial subscription
> > > > > > when creating the topic.
> > > > > > * If this field is not set, the initial subscription will not be
> > created.
> > > > > > *
> > > > > > * @param initialSubscriptionName
> > > > > > * Name of the initial subscription of the topic.
> > > > > > * @return the producer builder instance
> > > > > > */
> > > > > > ProducerBuilder<T> initialSubscriptionName(String
> > > > > initialSubscriptionName);
> > > > > > ```
> > > > > >
> > > > > >
> > > > > > ## Implementation
> > > > > >
> > > > > > When the deadLetterProducer is initialized, the consumer will set
> > the
> > > > > > initial subscription of the deadLetterProducer according to the
> > > > > > DeadLetterPolicy.
> > > > > >
> > > > > > When the broker creates a producer(after receiving the
> > > > > > CommandProducer), if it finds that the topic does not exist, it
> > will
> > > > > > not only create the topic(if the topic automatically creation is
> > > > > > enabled) but also create the initial subscription(if the initial
> > > > > > subscription name is set). When creating an initial subscription,
> > the
> > > > > > user needs to have the `canConsume` permission and the
> > > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > > >
> > > > > > ## Reject Alternatives
> > > > > >
> > > > > > ### Create the initial subscription using the consumer
> > > > > >
> > > > > > Before the deadLetterProducer is initialized, the consumer first
> > tries
> > > > > > to create a deadLetterConsumer using the initial subscription name
> > in
> > > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > > exception and create the deadLetterProducer.
> > > > > >
> > > > > > This is the original solution for this PIP. But this introduces too
> > > > > > much workload for the client. In the subsequent discussion, we try
> > to
> > > > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > > > closing the deadLetterConsumer, but again, this introduces
> > workload as
> > > > > > well as greater complexity.
> > > > > >
> > > > > > ### Use the retention policy to retent the data of the DLQ
> > > > > >
> > > > > > Before creating the deadLetterProducer, an admin can create a
> > > > > > retention policy for the topic or namespace. Then, consumers of the
> > > > > > topic have the duration of the retention policy to discover the
> > topic
> > > > > > and create a subscription before messages are lost.
> > > > > >
> > > > > > But users are not easy to set a different data retention policy or
> > > > > > create a new subscription for a lazy created DLQ topic.
> > > > > >
> > > > > > ### Add a new policy to retain data when the topic has no
> > subscriptions
> > > > > >
> > > > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > > > policy that makes it possible to retain messages indefinitely when
> > a
> > > > > > topic has no subscriptions.
> > > > > >
> > > > > > For the auto-created topic, the subscription can only be
> > determined at
> > > > > > the time of creation. It may or may not create. If users are able
> > to
> > > > > > determine which consumers are, and these consumers need to receive
> > any
> > > > > > message sent by the producer, they should create the topic and
> > > > > > subscription manually or use the consumer to trigger the topic
> > > > > > auto-creation, not the producer.
> > > > > >
> > > > > > ### Add the initial subscription name field to the metadata of the
> > > > > > CommandProducer
> > > > > >
> > > > > > Use producer metadata to carry the init subscription name so that
> > we
> > > > > > don't need to introduce a new field in CommandProducer, and the new
> > > > > > field looks a little confusing.
> > > > > >
> > > > > > While this does not change protobuf, it still implicitly changes
> > the
> > > > > > protocol. This solution has different advantages and disadvantages
> > > > > > than the current one. Finally, we decided to go with the current
> > > > > > solution.
> > > > > >
> > > > > > In addition, there is a discussion about whether we need to create
> > the
> > > > > > initial subscription on the retryLetterTopic. The consumer has
> > already
> > > > > > subscribed to the retryLetterTopic and created the subscription
> > before
> > > > > > creating the retryLetterProducer, so we don't need to create the
> > > > > > initial subscription for it.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Prototype implementation PR:
> > https://github.com/apache/pulsar/pull/13355
> > > > > >
> > > > > > ----
> > > > > >
> > > > > > Thanks
> > > > > > Zike Yang
> > > > >
> > > > >
> >
--
Zike Yang
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by PengHui Li <pe...@apache.org>.
+1 (binding)
Regards,
Penghui
On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mm...@apache.org>
wrote:
> +1 (non binding) - this proposal looks great! Thank you for a good
> discussion of this feature!
>
> Thanks,
> Michael
>
> On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
> >
> > +1 (binding)
> >
> > Thanks,
> > Hang
> >
> > Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> > >
> > > +1
> > >
> > >
> > >
> > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zkyang@streamnative.io
> .INVALID>
> > > > wrote:
> > > > >
> > > > > Hi Pulsar Community,
> > > > >
> > > > > This is the voting thread for PIP-124. It will stay open for at
> least
> > > > > 48 hours. Pasted below for quoting convenience.
> > > > >
> > > > > Here is the issue for this PIP:
> > > > https://github.com/apache/pulsar/issues/13408
> > > > >
> > > > > ----
> > > > >
> > > > > ## Motivation
> > > > >
> > > > > If we enable the DLQ when consuming messages. For some messages
> that
> > > > > can't be processed successfully, the messages will be moved to the
> > > > > DLQ, but if we do not specify the data retention for the namespace
> or
> > > > > create a subscription for the DLQ to retain the data, the data of
> the
> > > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > > initial subscription before sending messages to the DLQ.
> > > > >
> > > > > ## Goal
> > > > >
> > > > > Users can set the initial subscription name in the
> DeadLetterPolicy.
> > > > > The consumer will create the initial subscription before sending
> > > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > > the DLQ are not automatically deleted unexpectedly. If
> > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the
> initial
> > > > > subscription won't be created automatically. Otherwise, it will
> > > > > confuse the user. Users can explicitly create that subscription to
> > > > > handle this case.
> > > > >
> > > > > This PIP needs to be compatible with the previous behavior. The
> > > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > > default behavior is not to create the initial subscription which is
> > > > > consistent with the original behavior.
> > > > >
> > > > > ## API Changes
> > > > >
> > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > > >
> > > > > ```java
> > > > > /**
> > > > > * Name of the initial subscription name of the dead letter topic.
> > > > > * If this field is not set, the initial subscription will not be
> created.
> > > > > */
> > > > > private String initSubscriptionName;
> > > > > ```
> > > > >
> > > > > * Add a new field to the `CommandProducer`. The broker will use
> this
> > > > > field to create the initial subscription.
> > > > >
> > > > > ```proto
> > > > > optional string initial_subscription_name
> > > > > ```
> > > > >
> > > > > * Add a new config to the Producer Configuration. This allows the
> > > > > consumer to specify the initial subscription when creating the
> > > > > deadLetterProducer.
> > > > >
> > > > > ```java
> > > > > /**
> > > > > * Use this config to automatically create an initial subscription
> > > > > when creating the topic.
> > > > > * If this field is not set, the initial subscription will not be
> created.
> > > > > *
> > > > > * @param initialSubscriptionName
> > > > > * Name of the initial subscription of the topic.
> > > > > * @return the producer builder instance
> > > > > */
> > > > > ProducerBuilder<T> initialSubscriptionName(String
> > > > initialSubscriptionName);
> > > > > ```
> > > > >
> > > > >
> > > > > ## Implementation
> > > > >
> > > > > When the deadLetterProducer is initialized, the consumer will set
> the
> > > > > initial subscription of the deadLetterProducer according to the
> > > > > DeadLetterPolicy.
> > > > >
> > > > > When the broker creates a producer(after receiving the
> > > > > CommandProducer), if it finds that the topic does not exist, it
> will
> > > > > not only create the topic(if the topic automatically creation is
> > > > > enabled) but also create the initial subscription(if the initial
> > > > > subscription name is set). When creating an initial subscription,
> the
> > > > > user needs to have the `canConsume` permission and the
> > > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > > >
> > > > > ## Reject Alternatives
> > > > >
> > > > > ### Create the initial subscription using the consumer
> > > > >
> > > > > Before the deadLetterProducer is initialized, the consumer first
> tries
> > > > > to create a deadLetterConsumer using the initial subscription name
> in
> > > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > > exception and create the deadLetterProducer.
> > > > >
> > > > > This is the original solution for this PIP. But this introduces too
> > > > > much workload for the client. In the subsequent discussion, we try
> to
> > > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > > closing the deadLetterConsumer, but again, this introduces
> workload as
> > > > > well as greater complexity.
> > > > >
> > > > > ### Use the retention policy to retent the data of the DLQ
> > > > >
> > > > > Before creating the deadLetterProducer, an admin can create a
> > > > > retention policy for the topic or namespace. Then, consumers of the
> > > > > topic have the duration of the retention policy to discover the
> topic
> > > > > and create a subscription before messages are lost.
> > > > >
> > > > > But users are not easy to set a different data retention policy or
> > > > > create a new subscription for a lazy created DLQ topic.
> > > > >
> > > > > ### Add a new policy to retain data when the topic has no
> subscriptions
> > > > >
> > > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > > policy that makes it possible to retain messages indefinitely when
> a
> > > > > topic has no subscriptions.
> > > > >
> > > > > For the auto-created topic, the subscription can only be
> determined at
> > > > > the time of creation. It may or may not create. If users are able
> to
> > > > > determine which consumers are, and these consumers need to receive
> any
> > > > > message sent by the producer, they should create the topic and
> > > > > subscription manually or use the consumer to trigger the topic
> > > > > auto-creation, not the producer.
> > > > >
> > > > > ### Add the initial subscription name field to the metadata of the
> > > > > CommandProducer
> > > > >
> > > > > Use producer metadata to carry the init subscription name so that
> we
> > > > > don't need to introduce a new field in CommandProducer, and the new
> > > > > field looks a little confusing.
> > > > >
> > > > > While this does not change protobuf, it still implicitly changes
> the
> > > > > protocol. This solution has different advantages and disadvantages
> > > > > than the current one. Finally, we decided to go with the current
> > > > > solution.
> > > > >
> > > > > In addition, there is a discussion about whether we need to create
> the
> > > > > initial subscription on the retryLetterTopic. The consumer has
> already
> > > > > subscribed to the retryLetterTopic and created the subscription
> before
> > > > > creating the retryLetterProducer, so we don't need to create the
> > > > > initial subscription for it.
> > > > >
> > > > >
> > > > >
> > > > > Prototype implementation PR:
> https://github.com/apache/pulsar/pull/13355
> > > > >
> > > > > ----
> > > > >
> > > > > Thanks
> > > > > Zike Yang
> > > >
> > > >
>
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Michael Marshall <mm...@apache.org>.
+1 (non binding) - this proposal looks great! Thank you for a good
discussion of this feature!
Thanks,
Michael
On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <ch...@apache.org> wrote:
>
> +1 (binding)
>
> Thanks,
> Hang
>
> Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
> >
> > +1
> >
> >
> >
> > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zk...@streamnative.io.INVALID>
> > > wrote:
> > > >
> > > > Hi Pulsar Community,
> > > >
> > > > This is the voting thread for PIP-124. It will stay open for at least
> > > > 48 hours. Pasted below for quoting convenience.
> > > >
> > > > Here is the issue for this PIP:
> > > https://github.com/apache/pulsar/issues/13408
> > > >
> > > > ----
> > > >
> > > > ## Motivation
> > > >
> > > > If we enable the DLQ when consuming messages. For some messages that
> > > > can't be processed successfully, the messages will be moved to the
> > > > DLQ, but if we do not specify the data retention for the namespace or
> > > > create a subscription for the DLQ to retain the data, the data of the
> > > > DLQ will be removed automatically. Therefore, we need to create the
> > > > initial subscription before sending messages to the DLQ.
> > > >
> > > > ## Goal
> > > >
> > > > Users can set the initial subscription name in the DeadLetterPolicy.
> > > > The consumer will create the initial subscription before sending
> > > > messages to the DLQ. At this point, subsequent messages produced to
> > > > the DLQ are not automatically deleted unexpectedly. If
> > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > > > subscription won't be created automatically. Otherwise, it will
> > > > confuse the user. Users can explicitly create that subscription to
> > > > handle this case.
> > > >
> > > > This PIP needs to be compatible with the previous behavior. The
> > > > initial subscription name in the DeadLetterPolicy is optional. The
> > > > default behavior is not to create the initial subscription which is
> > > > consistent with the original behavior.
> > > >
> > > > ## API Changes
> > > >
> > > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > > >
> > > > ```java
> > > > /**
> > > > * Name of the initial subscription name of the dead letter topic.
> > > > * If this field is not set, the initial subscription will not be created.
> > > > */
> > > > private String initSubscriptionName;
> > > > ```
> > > >
> > > > * Add a new field to the `CommandProducer`. The broker will use this
> > > > field to create the initial subscription.
> > > >
> > > > ```proto
> > > > optional string initial_subscription_name
> > > > ```
> > > >
> > > > * Add a new config to the Producer Configuration. This allows the
> > > > consumer to specify the initial subscription when creating the
> > > > deadLetterProducer.
> > > >
> > > > ```java
> > > > /**
> > > > * Use this config to automatically create an initial subscription
> > > > when creating the topic.
> > > > * If this field is not set, the initial subscription will not be created.
> > > > *
> > > > * @param initialSubscriptionName
> > > > * Name of the initial subscription of the topic.
> > > > * @return the producer builder instance
> > > > */
> > > > ProducerBuilder<T> initialSubscriptionName(String
> > > initialSubscriptionName);
> > > > ```
> > > >
> > > >
> > > > ## Implementation
> > > >
> > > > When the deadLetterProducer is initialized, the consumer will set the
> > > > initial subscription of the deadLetterProducer according to the
> > > > DeadLetterPolicy.
> > > >
> > > > When the broker creates a producer(after receiving the
> > > > CommandProducer), if it finds that the topic does not exist, it will
> > > > not only create the topic(if the topic automatically creation is
> > > > enabled) but also create the initial subscription(if the initial
> > > > subscription name is set). When creating an initial subscription, the
> > > > user needs to have the `canConsume` permission and the
> > > > `allowAutoSubscriptionCreation` needs to be enabled.
> > > >
> > > > ## Reject Alternatives
> > > >
> > > > ### Create the initial subscription using the consumer
> > > >
> > > > Before the deadLetterProducer is initialized, the consumer first tries
> > > > to create a deadLetterConsumer using the initial subscription name in
> > > > the DeadLetterPolicy. When this subscription already exists, the
> > > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > > exception and create the deadLetterProducer.
> > > >
> > > > This is the original solution for this PIP. But this introduces too
> > > > much workload for the client. In the subsequent discussion, we try to
> > > > add a new command `CommandCreateSubscription` to avoid creating and
> > > > closing the deadLetterConsumer, but again, this introduces workload as
> > > > well as greater complexity.
> > > >
> > > > ### Use the retention policy to retent the data of the DLQ
> > > >
> > > > Before creating the deadLetterProducer, an admin can create a
> > > > retention policy for the topic or namespace. Then, consumers of the
> > > > topic have the duration of the retention policy to discover the topic
> > > > and create a subscription before messages are lost.
> > > >
> > > > But users are not easy to set a different data retention policy or
> > > > create a new subscription for a lazy created DLQ topic.
> > > >
> > > > ### Add a new policy to retain data when the topic has no subscriptions
> > > >
> > > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > > policy that makes it possible to retain messages indefinitely when a
> > > > topic has no subscriptions.
> > > >
> > > > For the auto-created topic, the subscription can only be determined at
> > > > the time of creation. It may or may not create. If users are able to
> > > > determine which consumers are, and these consumers need to receive any
> > > > message sent by the producer, they should create the topic and
> > > > subscription manually or use the consumer to trigger the topic
> > > > auto-creation, not the producer.
> > > >
> > > > ### Add the initial subscription name field to the metadata of the
> > > > CommandProducer
> > > >
> > > > Use producer metadata to carry the init subscription name so that we
> > > > don't need to introduce a new field in CommandProducer, and the new
> > > > field looks a little confusing.
> > > >
> > > > While this does not change protobuf, it still implicitly changes the
> > > > protocol. This solution has different advantages and disadvantages
> > > > than the current one. Finally, we decided to go with the current
> > > > solution.
> > > >
> > > > In addition, there is a discussion about whether we need to create the
> > > > initial subscription on the retryLetterTopic. The consumer has already
> > > > subscribed to the retryLetterTopic and created the subscription before
> > > > creating the retryLetterProducer, so we don't need to create the
> > > > initial subscription for it.
> > > >
> > > >
> > > >
> > > > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
> > > >
> > > > ----
> > > >
> > > > Thanks
> > > > Zike Yang
> > >
> > >
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Hang Chen <ch...@apache.org>.
+1 (binding)
Thanks,
Hang
Jia Zhai <zh...@apache.org> 于2022年1月26日周三 10:17写道：
>
> +1
>
>
>
> On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
> wrote:
>
> > +1 (non-binding)
> >
> > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zk...@streamnative.io.INVALID>
> > wrote:
> > >
> > > Hi Pulsar Community,
> > >
> > > This is the voting thread for PIP-124. It will stay open for at least
> > > 48 hours. Pasted below for quoting convenience.
> > >
> > > Here is the issue for this PIP:
> > https://github.com/apache/pulsar/issues/13408
> > >
> > > ----
> > >
> > > ## Motivation
> > >
> > > If we enable the DLQ when consuming messages. For some messages that
> > > can't be processed successfully, the messages will be moved to the
> > > DLQ, but if we do not specify the data retention for the namespace or
> > > create a subscription for the DLQ to retain the data, the data of the
> > > DLQ will be removed automatically. Therefore, we need to create the
> > > initial subscription before sending messages to the DLQ.
> > >
> > > ## Goal
> > >
> > > Users can set the initial subscription name in the DeadLetterPolicy.
> > > The consumer will create the initial subscription before sending
> > > messages to the DLQ. At this point, subsequent messages produced to
> > > the DLQ are not automatically deleted unexpectedly. If
> > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > > subscription won't be created automatically. Otherwise, it will
> > > confuse the user. Users can explicitly create that subscription to
> > > handle this case.
> > >
> > > This PIP needs to be compatible with the previous behavior. The
> > > initial subscription name in the DeadLetterPolicy is optional. The
> > > default behavior is not to create the initial subscription which is
> > > consistent with the original behavior.
> > >
> > > ## API Changes
> > >
> > > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> > >
> > > ```java
> > > /**
> > > * Name of the initial subscription name of the dead letter topic.
> > > * If this field is not set, the initial subscription will not be created.
> > > */
> > > private String initSubscriptionName;
> > > ```
> > >
> > > * Add a new field to the `CommandProducer`. The broker will use this
> > > field to create the initial subscription.
> > >
> > > ```proto
> > > optional string initial_subscription_name
> > > ```
> > >
> > > * Add a new config to the Producer Configuration. This allows the
> > > consumer to specify the initial subscription when creating the
> > > deadLetterProducer.
> > >
> > > ```java
> > > /**
> > > * Use this config to automatically create an initial subscription
> > > when creating the topic.
> > > * If this field is not set, the initial subscription will not be created.
> > > *
> > > * @param initialSubscriptionName
> > > * Name of the initial subscription of the topic.
> > > * @return the producer builder instance
> > > */
> > > ProducerBuilder<T> initialSubscriptionName(String
> > initialSubscriptionName);
> > > ```
> > >
> > >
> > > ## Implementation
> > >
> > > When the deadLetterProducer is initialized, the consumer will set the
> > > initial subscription of the deadLetterProducer according to the
> > > DeadLetterPolicy.
> > >
> > > When the broker creates a producer(after receiving the
> > > CommandProducer), if it finds that the topic does not exist, it will
> > > not only create the topic(if the topic automatically creation is
> > > enabled) but also create the initial subscription(if the initial
> > > subscription name is set). When creating an initial subscription, the
> > > user needs to have the `canConsume` permission and the
> > > `allowAutoSubscriptionCreation` needs to be enabled.
> > >
> > > ## Reject Alternatives
> > >
> > > ### Create the initial subscription using the consumer
> > >
> > > Before the deadLetterProducer is initialized, the consumer first tries
> > > to create a deadLetterConsumer using the initial subscription name in
> > > the DeadLetterPolicy. When this subscription already exists, the
> > > ConsumerBusy exception will occur. In this case, we can ignore that
> > > exception and create the deadLetterProducer.
> > >
> > > This is the original solution for this PIP. But this introduces too
> > > much workload for the client. In the subsequent discussion, we try to
> > > add a new command `CommandCreateSubscription` to avoid creating and
> > > closing the deadLetterConsumer, but again, this introduces workload as
> > > well as greater complexity.
> > >
> > > ### Use the retention policy to retent the data of the DLQ
> > >
> > > Before creating the deadLetterProducer, an admin can create a
> > > retention policy for the topic or namespace. Then, consumers of the
> > > topic have the duration of the retention policy to discover the topic
> > > and create a subscription before messages are lost.
> > >
> > > But users are not easy to set a different data retention policy or
> > > create a new subscription for a lazy created DLQ topic.
> > >
> > > ### Add a new policy to retain data when the topic has no subscriptions
> > >
> > > Adding a new policy for pulsar topics: a namespace or a topic level
> > > policy that makes it possible to retain messages indefinitely when a
> > > topic has no subscriptions.
> > >
> > > For the auto-created topic, the subscription can only be determined at
> > > the time of creation. It may or may not create. If users are able to
> > > determine which consumers are, and these consumers need to receive any
> > > message sent by the producer, they should create the topic and
> > > subscription manually or use the consumer to trigger the topic
> > > auto-creation, not the producer.
> > >
> > > ### Add the initial subscription name field to the metadata of the
> > > CommandProducer
> > >
> > > Use producer metadata to carry the init subscription name so that we
> > > don't need to introduce a new field in CommandProducer, and the new
> > > field looks a little confusing.
> > >
> > > While this does not change protobuf, it still implicitly changes the
> > > protocol. This solution has different advantages and disadvantages
> > > than the current one. Finally, we decided to go with the current
> > > solution.
> > >
> > > In addition, there is a discussion about whether we need to create the
> > > initial subscription on the retryLetterTopic. The consumer has already
> > > subscribed to the retryLetterTopic and created the subscription before
> > > creating the retryLetterProducer, so we don't need to create the
> > > initial subscription for it.
> > >
> > >
> > >
> > > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
> > >
> > > ----
> > >
> > > Thanks
> > > Zike Yang
> >
> >
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by Jia Zhai <zh...@apache.org>.
+1
On Tue, Jan 25, 2022 at 8:28 PM mattison chao <ma...@gmail.com>
wrote:
> +1 (non-binding)
>
> > On Jan 25, 2022, at 5:34 PM, Zike Yang <zk...@streamnative.io.INVALID>
> wrote:
> >
> > Hi Pulsar Community,
> >
> > This is the voting thread for PIP-124. It will stay open for at least
> > 48 hours. Pasted below for quoting convenience.
> >
> > Here is the issue for this PIP:
> https://github.com/apache/pulsar/issues/13408
> >
> > ----
> >
> > ## Motivation
> >
> > If we enable the DLQ when consuming messages. For some messages that
> > can't be processed successfully, the messages will be moved to the
> > DLQ, but if we do not specify the data retention for the namespace or
> > create a subscription for the DLQ to retain the data, the data of the
> > DLQ will be removed automatically. Therefore, we need to create the
> > initial subscription before sending messages to the DLQ.
> >
> > ## Goal
> >
> > Users can set the initial subscription name in the DeadLetterPolicy.
> > The consumer will create the initial subscription before sending
> > messages to the DLQ. At this point, subsequent messages produced to
> > the DLQ are not automatically deleted unexpectedly. If
> > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> > subscription won't be created automatically. Otherwise, it will
> > confuse the user. Users can explicitly create that subscription to
> > handle this case.
> >
> > This PIP needs to be compatible with the previous behavior. The
> > initial subscription name in the DeadLetterPolicy is optional. The
> > default behavior is not to create the initial subscription which is
> > consistent with the original behavior.
> >
> > ## API Changes
> >
> > * Add `initSubscriptionName` to the `DeadLetterPolicy`
> >
> > ```java
> > /**
> > * Name of the initial subscription name of the dead letter topic.
> > * If this field is not set, the initial subscription will not be created.
> > */
> > private String initSubscriptionName;
> > ```
> >
> > * Add a new field to the `CommandProducer`. The broker will use this
> > field to create the initial subscription.
> >
> > ```proto
> > optional string initial_subscription_name
> > ```
> >
> > * Add a new config to the Producer Configuration. This allows the
> > consumer to specify the initial subscription when creating the
> > deadLetterProducer.
> >
> > ```java
> > /**
> > * Use this config to automatically create an initial subscription
> > when creating the topic.
> > * If this field is not set, the initial subscription will not be created.
> > *
> > * @param initialSubscriptionName
> > * Name of the initial subscription of the topic.
> > * @return the producer builder instance
> > */
> > ProducerBuilder<T> initialSubscriptionName(String
> initialSubscriptionName);
> > ```
> >
> >
> > ## Implementation
> >
> > When the deadLetterProducer is initialized, the consumer will set the
> > initial subscription of the deadLetterProducer according to the
> > DeadLetterPolicy.
> >
> > When the broker creates a producer(after receiving the
> > CommandProducer), if it finds that the topic does not exist, it will
> > not only create the topic(if the topic automatically creation is
> > enabled) but also create the initial subscription(if the initial
> > subscription name is set). When creating an initial subscription, the
> > user needs to have the `canConsume` permission and the
> > `allowAutoSubscriptionCreation` needs to be enabled.
> >
> > ## Reject Alternatives
> >
> > ### Create the initial subscription using the consumer
> >
> > Before the deadLetterProducer is initialized, the consumer first tries
> > to create a deadLetterConsumer using the initial subscription name in
> > the DeadLetterPolicy. When this subscription already exists, the
> > ConsumerBusy exception will occur. In this case, we can ignore that
> > exception and create the deadLetterProducer.
> >
> > This is the original solution for this PIP. But this introduces too
> > much workload for the client. In the subsequent discussion, we try to
> > add a new command `CommandCreateSubscription` to avoid creating and
> > closing the deadLetterConsumer, but again, this introduces workload as
> > well as greater complexity.
> >
> > ### Use the retention policy to retent the data of the DLQ
> >
> > Before creating the deadLetterProducer, an admin can create a
> > retention policy for the topic or namespace. Then, consumers of the
> > topic have the duration of the retention policy to discover the topic
> > and create a subscription before messages are lost.
> >
> > But users are not easy to set a different data retention policy or
> > create a new subscription for a lazy created DLQ topic.
> >
> > ### Add a new policy to retain data when the topic has no subscriptions
> >
> > Adding a new policy for pulsar topics: a namespace or a topic level
> > policy that makes it possible to retain messages indefinitely when a
> > topic has no subscriptions.
> >
> > For the auto-created topic, the subscription can only be determined at
> > the time of creation. It may or may not create. If users are able to
> > determine which consumers are, and these consumers need to receive any
> > message sent by the producer, they should create the topic and
> > subscription manually or use the consumer to trigger the topic
> > auto-creation, not the producer.
> >
> > ### Add the initial subscription name field to the metadata of the
> > CommandProducer
> >
> > Use producer metadata to carry the init subscription name so that we
> > don't need to introduce a new field in CommandProducer, and the new
> > field looks a little confusing.
> >
> > While this does not change protobuf, it still implicitly changes the
> > protocol. This solution has different advantages and disadvantages
> > than the current one. Finally, we decided to go with the current
> > solution.
> >
> > In addition, there is a discussion about whether we need to create the
> > initial subscription on the retryLetterTopic. The consumer has already
> > subscribed to the retryLetterTopic and created the subscription before
> > creating the retryLetterProducer, so we don't need to create the
> > initial subscription for it.
> >
> >
> >
> > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
> >
> > ----
> >
> > Thanks
> > Zike Yang
>
>
Re: [VOTE] PIP-124: Create init subscription before sending message to DLQ
Posted by mattison chao <ma...@gmail.com>.
+1 (non-binding)
> On Jan 25, 2022, at 5:34 PM, Zike Yang <zk...@streamnative.io.INVALID> wrote:
>
> Hi Pulsar Community,
>
> This is the voting thread for PIP-124. It will stay open for at least
> 48 hours. Pasted below for quoting convenience.
>
> Here is the issue for this PIP: https://github.com/apache/pulsar/issues/13408
>
> ----
>
> ## Motivation
>
> If we enable the DLQ when consuming messages. For some messages that
> can't be processed successfully, the messages will be moved to the
> DLQ, but if we do not specify the data retention for the namespace or
> create a subscription for the DLQ to retain the data, the data of the
> DLQ will be removed automatically. Therefore, we need to create the
> initial subscription before sending messages to the DLQ.
>
> ## Goal
>
> Users can set the initial subscription name in the DeadLetterPolicy.
> The consumer will create the initial subscription before sending
> messages to the DLQ. At this point, subsequent messages produced to
> the DLQ are not automatically deleted unexpectedly. If
> `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial
> subscription won't be created automatically. Otherwise, it will
> confuse the user. Users can explicitly create that subscription to
> handle this case.
>
> This PIP needs to be compatible with the previous behavior. The
> initial subscription name in the DeadLetterPolicy is optional. The
> default behavior is not to create the initial subscription which is
> consistent with the original behavior.
>
> ## API Changes
>
> * Add `initSubscriptionName` to the `DeadLetterPolicy`
>
> ```java
> /**
> * Name of the initial subscription name of the dead letter topic.
> * If this field is not set, the initial subscription will not be created.
> */
> private String initSubscriptionName;
> ```
>
> * Add a new field to the `CommandProducer`. The broker will use this
> field to create the initial subscription.
>
> ```proto
> optional string initial_subscription_name
> ```
>
> * Add a new config to the Producer Configuration. This allows the
> consumer to specify the initial subscription when creating the
> deadLetterProducer.
>
> ```java
> /**
> * Use this config to automatically create an initial subscription
> when creating the topic.
> * If this field is not set, the initial subscription will not be created.
> *
> * @param initialSubscriptionName
> * Name of the initial subscription of the topic.
> * @return the producer builder instance
> */
> ProducerBuilder<T> initialSubscriptionName(String initialSubscriptionName);
> ```
>
>
> ## Implementation
>
> When the deadLetterProducer is initialized, the consumer will set the
> initial subscription of the deadLetterProducer according to the
> DeadLetterPolicy.
>
> When the broker creates a producer(after receiving the
> CommandProducer), if it finds that the topic does not exist, it will
> not only create the topic(if the topic automatically creation is
> enabled) but also create the initial subscription(if the initial
> subscription name is set). When creating an initial subscription, the
> user needs to have the `canConsume` permission and the
> `allowAutoSubscriptionCreation` needs to be enabled.
>
> ## Reject Alternatives
>
> ### Create the initial subscription using the consumer
>
> Before the deadLetterProducer is initialized, the consumer first tries
> to create a deadLetterConsumer using the initial subscription name in
> the DeadLetterPolicy. When this subscription already exists, the
> ConsumerBusy exception will occur. In this case, we can ignore that
> exception and create the deadLetterProducer.
>
> This is the original solution for this PIP. But this introduces too
> much workload for the client. In the subsequent discussion, we try to
> add a new command `CommandCreateSubscription` to avoid creating and
> closing the deadLetterConsumer, but again, this introduces workload as
> well as greater complexity.
>
> ### Use the retention policy to retent the data of the DLQ
>
> Before creating the deadLetterProducer, an admin can create a
> retention policy for the topic or namespace. Then, consumers of the
> topic have the duration of the retention policy to discover the topic
> and create a subscription before messages are lost.
>
> But users are not easy to set a different data retention policy or
> create a new subscription for a lazy created DLQ topic.
>
> ### Add a new policy to retain data when the topic has no subscriptions
>
> Adding a new policy for pulsar topics: a namespace or a topic level
> policy that makes it possible to retain messages indefinitely when a
> topic has no subscriptions.
>
> For the auto-created topic, the subscription can only be determined at
> the time of creation. It may or may not create. If users are able to
> determine which consumers are, and these consumers need to receive any
> message sent by the producer, they should create the topic and
> subscription manually or use the consumer to trigger the topic
> auto-creation, not the producer.
>
> ### Add the initial subscription name field to the metadata of the
> CommandProducer
>
> Use producer metadata to carry the init subscription name so that we
> don't need to introduce a new field in CommandProducer, and the new
> field looks a little confusing.
>
> While this does not change protobuf, it still implicitly changes the
> protocol. This solution has different advantages and disadvantages
> than the current one. Finally, we decided to go with the current
> solution.
>
> In addition, there is a discussion about whether we need to create the
> initial subscription on the retryLetterTopic. The consumer has already
> subscribed to the retryLetterTopic and created the subscription before
> creating the retryLetterProducer, so we don't need to create the
> initial subscription for it.
>
>
>
> Prototype implementation PR: https://github.com/apache/pulsar/pull/13355
>
> ----
>
> Thanks
> Zike Yang