You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by 丛搏 <co...@gmail.com> on 2022/01/13 08:32:10 UTC
[DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
Hi :
issue link : https://github.com/streamnative/support-tickets/issues/408 <https://github.com/streamnative/support-tickets/issues/408>
# Background
Now we have many clear namespace backlog interfaces etc.
```
clearNamespaceBacklog(String namespace)
clearNamespaceBacklogAsync(String namespace)
clearNamespaceBacklogForSubscription(String namespace, String subscription)
clearNamespaceBacklogForSubscriptionAsync(String namespace, String subscription)
clearNamespaceBundleBacklog(String namespace, String bundle)
clearNamespaceBundleBacklogAsync(String namespace, String bundle)
clearNamespaceBundleBacklogForSubscription(String namespace, String bundle, String subscription)
clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String bundle, String subscription)
```
There are two types of interfaces:
1. clear namespace backlog with subscriptionName
2. clear namespace backlog no subscriptionName
But there have a problem, users do not know that there are many system topics in the namespace.
```
/**
* Local topic name for the namespace events.
*/
public static final String NAMESPACE_EVENTS_LOCAL_NAME = "__change_events";
/**
* Local topic name for the transaction buffer snapshot.
*/
public static final String TRANSACTION_BUFFER_SNAPSHOT = "__transaction_buffer_snapshot";
public static final String SUBSCRIPTION_NAME = "transaction-buffer-sub";
```
User only want to clear the backlog witch they own created and sub. But now we have clear the system topic sub backlog when user don't specify subName and clear system sub with a topic witch user created. It will bring many problems. etc.
1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this namespace, but user don't know and they don't want to be cleared. If the user does not understand the problems caused by this interface, it will delete many topic policy configurations and is difficult to recover, which will lead to unpredictable and disastrous problems.
2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause transaction buffer recover incompletely.
3. clear `transaction-buffer-sub` sub also will cause transaction buffer recover incompletely.
4. if add new system topic and system sub, we should think about this problem.
# Motivation
We want to solve the above problems, but we can't make users unable to delete the backlog of system topic and system sub.
So, we should solve types of interface
There are two types of interfaces:
1. clear namespace backlog with subscriptionName
2. clear namespace backlog no subscriptionName
when clear namespace backlog no subscriptionName:
It represents user know the sub name is system subName and really want to delete it, We should not block the user clear system sub backlog and do not allow the user to create system sub. If the user uses it, it means that the user has considered the result of the clear system sub backlog
when clear namespace backlog with subscriptionName:
Because the user may not consider the existence of system topic and system sub, and the wrong clear backlog may lead to serious results. Therefore, this interface cannot delete the backlog of system topic and the backlog of system sub. If the user really wants to clear backlog, please bring subscriptionName and use another type of interface with subscriptionName.
**Clear namespace and clear namespace bundle all need to handle is logical.**
# implement
```
class SystemTopicName {
/**
* Local topic name for the namespace events.
*/
public static final String NAMESPACE_EVENTS_LOCAL_NAME = "__change_events";
/**
* Local topic name for the transaction buffer snapshot.
*/
public static final String TRANSACTION_BUFFER_SNAPSHOT = "__transaction_buffer_snapshot";
}
```
```
class SystemSubName {
public static final String SUBSCRIPTION_NAME = "transaction-buffer-sub";
}
```
1. In order to make the code easier to maintain and reduce redundancy, we need to move systemTopic and systemSub to a specific classes to manage.
2. If the newly added systemTopic and systemSub are created from this class, unpredictable problems can be prevented
Let me know if you have different ideas or suggestions！！
Thanks!
Bo Cong
Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
Posted by Michael Marshall <mm...@apache.org>.
> Currently, we don't have some system topics that users can consume, I think
> it's a good chance to normalise the system topic naming rule.
> but "__" is more common, we need more complex naming rules to reduce
> restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
> etc.
If "__" is too common, I am fine with something more unique. Ideally,
it won't be too long, though.
> Others, some plugins like "Protocol handler" want to use system topics or
> pulsar-manager can monitor how many system topics we have.
It'd be helpful to enumerate our current use of system topics. I
imagine that we'll continue to add system topics to Pulsar,
which is why I want a comprehensive design and definition for them.
> I think we also need to support registering system topic names, because currently
> we can only rely on system topic names to create system topics. (Maybe changing
> the internal api to support creating system topic would also be a good
> option. But then it's harder to monitor).
Registering individual topics as system topics on the fly will not
allow us to protect against name collisions with user topics and
system topics. For this reason, we already prevent users from creating
transaction system topics here [0]. By having a generic prefix, we'll
be able to trivially protect all system topic names.
Here are some of my current thoughts on system topics:
A system topic is a topic that is completely internal to Pulsar
components. Internally, it is a normal topic. It requires elevated
permission to produce/consume when authorization is enabled, even if
topic level policies are not enabled. Generic calls like
`clearNamespaceBacklog` should not affect system topics. Deleting a
namespace or tenant should delete the system topics within it.
Let me know what you think.
Thanks,
Michael
[0] https://github.com/apache/pulsar/blob/35f0e13fc3385b54e88ddd8e62e44146cf3b060d/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L1341-L1347
On Sun, Jan 23, 2022 at 7:23 AM mattison chao <ma...@gmail.com> wrote:
>
> HI, Michael Marshall
>
> > I am concerned that this design will not scale and that it will be very
> expensive on installations with many topics.
>
> I very much agree with your concern, but currently we already have some
> topic names to use ``endWith()`` and ``startWith()`` to determine.
>
> > It would also improve the speed of operations on namespaces because
> filtering out system topics would only require a quick check for
> `startsWith("__")`.
>
> Currently, we don't have some system topics that users can consume, I think
> it's a good chance to normalise the system topic naming rule.
> but "__" is more common, we need more complex naming rules to reduce
> restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
> etc.
>
> Others, some plugins like "Protocol handler" want to use system topics or
> pulsar-manager can monitor how many system topics we have. I think we also
> need to support registering system topic names, because currently we can
> only rely on system topic names to create system topics. (Maybe changing
> the internal api to support creating system topic would also be a good
> option. But then it's harder to monitor).
>
> Do you have any great ideas ?
>
> Best,
> Mattison
>
> On Sun, 23 Jan 2022 at 14:46, Michael Marshall <mm...@apache.org> wrote:
>
> > This is a really important thread. I think our system topic design
> > needs more definition, structure, and documentation. Thanks for
> > starting the thread, Bo Cong.
> >
> > > If the user uses it, it means that the user has considered the result of
> > the clear system sub backlog
> >
> > I think we need to discuss authorization for operating on system
> > topics, especially when topic level policies are not enabled.
> >
> > From reading the PIP [0], it looks like we already have many different
> > ways to designate a system topic (prefixes, suffixes, and total string
> > equality). I am concerned that this design will not scale and that it
> > will be very expensive on installations with many topics.
> >
> > Is it possible for us to pivot our implementation so that system
> > topics and only system topics start with two underscores? This could
> > simplify system topic authorization. It would also improve the speed
> > of operations on namespaces because filtering out system topics would
> > only require a quick check for `startsWith("__")`.
> >
> > While it's possible that users already have topics that start with
> > "__", I think it is unlikely, and I think it is essential that we
> > design system topics so that they are highly scalable.
> >
> > Thanks,
> > Michael
> >
> > [0] https://github.com/apache/pulsar/issues/13794
> >
> >
> > On Thu, Jan 13, 2022 at 10:27 AM PengHui Li <pe...@apache.org> wrote:
> > >
> > > I agree with the point, we should avoid the `clearNamespaceBacklog(String
> > > namespace)` to apply to the internal topic or internal cursor.
> > > It will introduce abnormal behaviors, e.g. clear a replication cursor
> > > backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
> > >
> > > I would suggest letting the `clearNamespaceBacklog(String namespace)`
> > > method does not apply to the Pulsar internal topics and cursors.
> > >
> > > Thanks,
> > > Penghui
> > >
> > > On Thu, Jan 13, 2022 at 4:32 PM 丛搏 <co...@gmail.com> wrote:
> > >
> > > > Hi :
> > > >
> > > > issue link :
> > https://github.com/streamnative/support-tickets/issues/408 <
> > > > https://github.com/streamnative/support-tickets/issues/408>
> > > >
> > > > # Background
> > > > Now we have many clear namespace backlog interfaces etc.
> > > > ```
> > > > clearNamespaceBacklog(String namespace)
> > > >
> > > > clearNamespaceBacklogAsync(String namespace)
> > > >
> > > > clearNamespaceBacklogForSubscription(String namespace, String
> > subscription)
> > > >
> > > > clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> > > > subscription)
> > > >
> > > > clearNamespaceBundleBacklog(String namespace, String bundle)
> > > >
> > > > clearNamespaceBundleBacklogAsync(String namespace, String bundle)
> > > >
> > > > clearNamespaceBundleBacklogForSubscription(String namespace, String
> > > > bundle, String subscription)
> > > >
> > > > clearNamespaceBundleBacklogForSubscriptionAsync(String namespace,
> > String
> > > > bundle, String subscription)
> > > > ```
> > > > There are two types of interfaces:
> > > > 1. clear namespace backlog with subscriptionName
> > > > 2. clear namespace backlog no subscriptionName
> > > >
> > > > But there have a problem, users do not know that there are many system
> > > > topics in the namespace.
> > > >
> > > > ```
> > > >
> > > > /**
> > > > * Local topic name for the namespace events.
> > > > */
> > > > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > > > "__change_events";
> > > >
> > > > /**
> > > > * Local topic name for the transaction buffer snapshot.
> > > > */
> > > > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > > > "__transaction_buffer_snapshot";
> > > >
> > > > public static final String SUBSCRIPTION_NAME =
> > > > "transaction-buffer-sub";
> > > > ```
> > > > User only want to clear the backlog witch they own created and sub. But
> > > > now we have clear the system topic sub backlog when user don't specify
> > > > subName and clear system sub with a topic witch user created. It will
> > bring
> > > > many problems. etc.
> > > > 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in
> > this
> > > > namespace, but user don't know and they don't want to be cleared. If
> > the
> > > > user does not understand the problems caused by this interface, it will
> > > > delete many topic policy configurations and is difficult to recover,
> > which
> > > > will lead to unpredictable and disastrous problems.
> > > > 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> > > > transaction buffer recover incompletely.
> > > > 3. clear `transaction-buffer-sub` sub also will cause transaction
> > buffer
> > > > recover incompletely.
> > > > 4. if add new system topic and system sub, we should think about this
> > > > problem.
> > > >
> > > > # Motivation
> > > > We want to solve the above problems, but we can't make users unable to
> > > > delete the backlog of system topic and system sub.
> > > > So, we should solve types of interface
> > > > There are two types of interfaces:
> > > > 1. clear namespace backlog with subscriptionName
> > > > 2. clear namespace backlog no subscriptionName
> > > >
> > > > when clear namespace backlog no subscriptionName:
> > > > It represents user know the sub name is system subName and really want
> > to
> > > > delete it, We should not block the user clear system sub backlog and
> > do not
> > > > allow the user to create system sub. If the user uses it, it means
> > that the
> > > > user has considered the result of the clear system sub backlog
> > > >
> > > > when clear namespace backlog with subscriptionName:
> > > > Because the user may not consider the existence of system topic and
> > system
> > > > sub, and the wrong clear backlog may lead to serious results.
> > Therefore,
> > > > this interface cannot delete the backlog of system topic and the
> > backlog of
> > > > system sub. If the user really wants to clear backlog, please bring
> > > > subscriptionName and use another type of interface with
> > subscriptionName.
> > > >
> > > > **Clear namespace and clear namespace bundle all need to handle is
> > > > logical.**
> > > >
> > > > # implement
> > > >
> > > > ```
> > > > class SystemTopicName {
> > > > /**
> > > > * Local topic name for the namespace events.
> > > > */
> > > > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > > > "__change_events";
> > > >
> > > > /**
> > > > * Local topic name for the transaction buffer snapshot.
> > > > */
> > > > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > > > "__transaction_buffer_snapshot";
> > > > }
> > > > ```
> > > > ```
> > > > class SystemSubName {
> > > >
> > > > public static final String SUBSCRIPTION_NAME =
> > > > "transaction-buffer-sub";
> > > >
> > > > }
> > > > ```
> > > >
> > > > 1. In order to make the code easier to maintain and reduce redundancy,
> > we
> > > > need to move systemTopic and systemSub to a specific classes to manage.
> > > > 2. If the newly added systemTopic and systemSub are created from this
> > > > class, unpredictable problems can be prevented
> > > >
> > > >
> > > >
> > > > Let me know if you have different ideas or suggestions！！
> > > >
> > > > Thanks!
> > > > Bo Cong
> >
Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
Posted by mattison chao <ma...@gmail.com>.
HI, Michael Marshall
> I am concerned that this design will not scale and that it will be very
expensive on installations with many topics.
I very much agree with your concern, but currently we already have some
topic names to use ``endWith()`` and ``startWith()`` to determine.
> It would also improve the speed of operations on namespaces because
filtering out system topics would only require a quick check for
`startsWith("__")`.
Currently, we don't have some system topics that users can consume, I think
it's a good chance to normalise the system topic naming rule.
but "__" is more common, we need more complex naming rules to reduce
restrictions on users. Maybe we can use some prefix like "__pulsar_sys_"
etc.
Others, some plugins like "Protocol handler" want to use system topics or
pulsar-manager can monitor how many system topics we have. I think we also
need to support registering system topic names, because currently we can
only rely on system topic names to create system topics. (Maybe changing
the internal api to support creating system topic would also be a good
option. But then it's harder to monitor).
Do you have any great ideas ?
Best,
Mattison
On Sun, 23 Jan 2022 at 14:46, Michael Marshall <mm...@apache.org> wrote:
> This is a really important thread. I think our system topic design
> needs more definition, structure, and documentation. Thanks for
> starting the thread, Bo Cong.
>
> > If the user uses it, it means that the user has considered the result of
> the clear system sub backlog
>
> I think we need to discuss authorization for operating on system
> topics, especially when topic level policies are not enabled.
>
> From reading the PIP [0], it looks like we already have many different
> ways to designate a system topic (prefixes, suffixes, and total string
> equality). I am concerned that this design will not scale and that it
> will be very expensive on installations with many topics.
>
> Is it possible for us to pivot our implementation so that system
> topics and only system topics start with two underscores? This could
> simplify system topic authorization. It would also improve the speed
> of operations on namespaces because filtering out system topics would
> only require a quick check for `startsWith("__")`.
>
> While it's possible that users already have topics that start with
> "__", I think it is unlikely, and I think it is essential that we
> design system topics so that they are highly scalable.
>
> Thanks,
> Michael
>
> [0] https://github.com/apache/pulsar/issues/13794
>
>
> On Thu, Jan 13, 2022 at 10:27 AM PengHui Li <pe...@apache.org> wrote:
> >
> > I agree with the point, we should avoid the `clearNamespaceBacklog(String
> > namespace)` to apply to the internal topic or internal cursor.
> > It will introduce abnormal behaviors, e.g. clear a replication cursor
> > backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
> >
> > I would suggest letting the `clearNamespaceBacklog(String namespace)`
> > method does not apply to the Pulsar internal topics and cursors.
> >
> > Thanks,
> > Penghui
> >
> > On Thu, Jan 13, 2022 at 4:32 PM 丛搏 <co...@gmail.com> wrote:
> >
> > > Hi :
> > >
> > > issue link :
> https://github.com/streamnative/support-tickets/issues/408 <
> > > https://github.com/streamnative/support-tickets/issues/408>
> > >
> > > # Background
> > > Now we have many clear namespace backlog interfaces etc.
> > > ```
> > > clearNamespaceBacklog(String namespace)
> > >
> > > clearNamespaceBacklogAsync(String namespace)
> > >
> > > clearNamespaceBacklogForSubscription(String namespace, String
> subscription)
> > >
> > > clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> > > subscription)
> > >
> > > clearNamespaceBundleBacklog(String namespace, String bundle)
> > >
> > > clearNamespaceBundleBacklogAsync(String namespace, String bundle)
> > >
> > > clearNamespaceBundleBacklogForSubscription(String namespace, String
> > > bundle, String subscription)
> > >
> > > clearNamespaceBundleBacklogForSubscriptionAsync(String namespace,
> String
> > > bundle, String subscription)
> > > ```
> > > There are two types of interfaces:
> > > 1. clear namespace backlog with subscriptionName
> > > 2. clear namespace backlog no subscriptionName
> > >
> > > But there have a problem, users do not know that there are many system
> > > topics in the namespace.
> > >
> > > ```
> > >
> > > /**
> > > * Local topic name for the namespace events.
> > > */
> > > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > > "__change_events";
> > >
> > > /**
> > > * Local topic name for the transaction buffer snapshot.
> > > */
> > > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > > "__transaction_buffer_snapshot";
> > >
> > > public static final String SUBSCRIPTION_NAME =
> > > "transaction-buffer-sub";
> > > ```
> > > User only want to clear the backlog witch they own created and sub. But
> > > now we have clear the system topic sub backlog when user don't specify
> > > subName and clear system sub with a topic witch user created. It will
> bring
> > > many problems. etc.
> > > 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in
> this
> > > namespace, but user don't know and they don't want to be cleared. If
> the
> > > user does not understand the problems caused by this interface, it will
> > > delete many topic policy configurations and is difficult to recover,
> which
> > > will lead to unpredictable and disastrous problems.
> > > 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> > > transaction buffer recover incompletely.
> > > 3. clear `transaction-buffer-sub` sub also will cause transaction
> buffer
> > > recover incompletely.
> > > 4. if add new system topic and system sub, we should think about this
> > > problem.
> > >
> > > # Motivation
> > > We want to solve the above problems, but we can't make users unable to
> > > delete the backlog of system topic and system sub.
> > > So, we should solve types of interface
> > > There are two types of interfaces:
> > > 1. clear namespace backlog with subscriptionName
> > > 2. clear namespace backlog no subscriptionName
> > >
> > > when clear namespace backlog no subscriptionName:
> > > It represents user know the sub name is system subName and really want
> to
> > > delete it, We should not block the user clear system sub backlog and
> do not
> > > allow the user to create system sub. If the user uses it, it means
> that the
> > > user has considered the result of the clear system sub backlog
> > >
> > > when clear namespace backlog with subscriptionName:
> > > Because the user may not consider the existence of system topic and
> system
> > > sub, and the wrong clear backlog may lead to serious results.
> Therefore,
> > > this interface cannot delete the backlog of system topic and the
> backlog of
> > > system sub. If the user really wants to clear backlog, please bring
> > > subscriptionName and use another type of interface with
> subscriptionName.
> > >
> > > **Clear namespace and clear namespace bundle all need to handle is
> > > logical.**
> > >
> > > # implement
> > >
> > > ```
> > > class SystemTopicName {
> > > /**
> > > * Local topic name for the namespace events.
> > > */
> > > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > > "__change_events";
> > >
> > > /**
> > > * Local topic name for the transaction buffer snapshot.
> > > */
> > > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > > "__transaction_buffer_snapshot";
> > > }
> > > ```
> > > ```
> > > class SystemSubName {
> > >
> > > public static final String SUBSCRIPTION_NAME =
> > > "transaction-buffer-sub";
> > >
> > > }
> > > ```
> > >
> > > 1. In order to make the code easier to maintain and reduce redundancy,
> we
> > > need to move systemTopic and systemSub to a specific classes to manage.
> > > 2. If the newly added systemTopic and systemSub are created from this
> > > class, unpredictable problems can be prevented
> > >
> > >
> > >
> > > Let me know if you have different ideas or suggestions！！
> > >
> > > Thanks!
> > > Bo Cong
>
Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
Posted by Michael Marshall <mm...@apache.org>.
This is a really important thread. I think our system topic design
needs more definition, structure, and documentation. Thanks for
starting the thread, Bo Cong.
> If the user uses it, it means that the user has considered the result of the clear system sub backlog
I think we need to discuss authorization for operating on system
topics, especially when topic level policies are not enabled.
From reading the PIP [0], it looks like we already have many different
ways to designate a system topic (prefixes, suffixes, and total string
equality). I am concerned that this design will not scale and that it
will be very expensive on installations with many topics.
Is it possible for us to pivot our implementation so that system
topics and only system topics start with two underscores? This could
simplify system topic authorization. It would also improve the speed
of operations on namespaces because filtering out system topics would
only require a quick check for `startsWith("__")`.
While it's possible that users already have topics that start with
"__", I think it is unlikely, and I think it is essential that we
design system topics so that they are highly scalable.
Thanks,
Michael
[0] https://github.com/apache/pulsar/issues/13794
On Thu, Jan 13, 2022 at 10:27 AM PengHui Li <pe...@apache.org> wrote:
>
> I agree with the point, we should avoid the `clearNamespaceBacklog(String
> namespace)` to apply to the internal topic or internal cursor.
> It will introduce abnormal behaviors, e.g. clear a replication cursor
> backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
>
> I would suggest letting the `clearNamespaceBacklog(String namespace)`
> method does not apply to the Pulsar internal topics and cursors.
>
> Thanks,
> Penghui
>
> On Thu, Jan 13, 2022 at 4:32 PM 丛搏 <co...@gmail.com> wrote:
>
> > Hi :
> >
> > issue link : https://github.com/streamnative/support-tickets/issues/408 <
> > https://github.com/streamnative/support-tickets/issues/408>
> >
> > # Background
> > Now we have many clear namespace backlog interfaces etc.
> > ```
> > clearNamespaceBacklog(String namespace)
> >
> > clearNamespaceBacklogAsync(String namespace)
> >
> > clearNamespaceBacklogForSubscription(String namespace, String subscription)
> >
> > clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> > subscription)
> >
> > clearNamespaceBundleBacklog(String namespace, String bundle)
> >
> > clearNamespaceBundleBacklogAsync(String namespace, String bundle)
> >
> > clearNamespaceBundleBacklogForSubscription(String namespace, String
> > bundle, String subscription)
> >
> > clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String
> > bundle, String subscription)
> > ```
> > There are two types of interfaces:
> > 1. clear namespace backlog with subscriptionName
> > 2. clear namespace backlog no subscriptionName
> >
> > But there have a problem, users do not know that there are many system
> > topics in the namespace.
> >
> > ```
> >
> > /**
> > * Local topic name for the namespace events.
> > */
> > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > "__change_events";
> >
> > /**
> > * Local topic name for the transaction buffer snapshot.
> > */
> > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > "__transaction_buffer_snapshot";
> >
> > public static final String SUBSCRIPTION_NAME =
> > "transaction-buffer-sub";
> > ```
> > User only want to clear the backlog witch they own created and sub. But
> > now we have clear the system topic sub backlog when user don't specify
> > subName and clear system sub with a topic witch user created. It will bring
> > many problems. etc.
> > 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this
> > namespace, but user don't know and they don't want to be cleared. If the
> > user does not understand the problems caused by this interface, it will
> > delete many topic policy configurations and is difficult to recover, which
> > will lead to unpredictable and disastrous problems.
> > 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> > transaction buffer recover incompletely.
> > 3. clear `transaction-buffer-sub` sub also will cause transaction buffer
> > recover incompletely.
> > 4. if add new system topic and system sub, we should think about this
> > problem.
> >
> > # Motivation
> > We want to solve the above problems, but we can't make users unable to
> > delete the backlog of system topic and system sub.
> > So, we should solve types of interface
> > There are two types of interfaces:
> > 1. clear namespace backlog with subscriptionName
> > 2. clear namespace backlog no subscriptionName
> >
> > when clear namespace backlog no subscriptionName:
> > It represents user know the sub name is system subName and really want to
> > delete it, We should not block the user clear system sub backlog and do not
> > allow the user to create system sub. If the user uses it, it means that the
> > user has considered the result of the clear system sub backlog
> >
> > when clear namespace backlog with subscriptionName:
> > Because the user may not consider the existence of system topic and system
> > sub, and the wrong clear backlog may lead to serious results. Therefore,
> > this interface cannot delete the backlog of system topic and the backlog of
> > system sub. If the user really wants to clear backlog, please bring
> > subscriptionName and use another type of interface with subscriptionName.
> >
> > **Clear namespace and clear namespace bundle all need to handle is
> > logical.**
> >
> > # implement
> >
> > ```
> > class SystemTopicName {
> > /**
> > * Local topic name for the namespace events.
> > */
> > public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> > "__change_events";
> >
> > /**
> > * Local topic name for the transaction buffer snapshot.
> > */
> > public static final String TRANSACTION_BUFFER_SNAPSHOT =
> > "__transaction_buffer_snapshot";
> > }
> > ```
> > ```
> > class SystemSubName {
> >
> > public static final String SUBSCRIPTION_NAME =
> > "transaction-buffer-sub";
> >
> > }
> > ```
> >
> > 1. In order to make the code easier to maintain and reduce redundancy, we
> > need to move systemTopic and systemSub to a specific classes to manage.
> > 2. If the newly added systemTopic and systemSub are created from this
> > class, unpredictable problems can be prevented
> >
> >
> >
> > Let me know if you have different ideas or suggestions！！
> >
> > Thanks!
> > Bo Cong
Re: [DISCUSS] [Transaction] Clear namespace backlog can't clear system topic and system sub backlog.
Posted by PengHui Li <pe...@apache.org>.
I agree with the point, we should avoid the `clearNamespaceBacklog(String
namespace)` to apply to the internal topic or internal cursor.
It will introduce abnormal behaviors, e.g. clear a replication cursor
backlog, clear a dedup cursor backlog, clear a compaction cursor backlog.
I would suggest letting the `clearNamespaceBacklog(String namespace)`
method does not apply to the Pulsar internal topics and cursors.
Thanks,
Penghui
On Thu, Jan 13, 2022 at 4:32 PM 丛搏 <co...@gmail.com> wrote:
> Hi :
>
> issue link : https://github.com/streamnative/support-tickets/issues/408 <
> https://github.com/streamnative/support-tickets/issues/408>
>
> # Background
> Now we have many clear namespace backlog interfaces etc.
> ```
> clearNamespaceBacklog(String namespace)
>
> clearNamespaceBacklogAsync(String namespace)
>
> clearNamespaceBacklogForSubscription(String namespace, String subscription)
>
> clearNamespaceBacklogForSubscriptionAsync(String namespace, String
> subscription)
>
> clearNamespaceBundleBacklog(String namespace, String bundle)
>
> clearNamespaceBundleBacklogAsync(String namespace, String bundle)
>
> clearNamespaceBundleBacklogForSubscription(String namespace, String
> bundle, String subscription)
>
> clearNamespaceBundleBacklogForSubscriptionAsync(String namespace, String
> bundle, String subscription)
> ```
> There are two types of interfaces:
> 1. clear namespace backlog with subscriptionName
> 2. clear namespace backlog no subscriptionName
>
> But there have a problem, users do not know that there are many system
> topics in the namespace.
>
> ```
>
> /**
> * Local topic name for the namespace events.
> */
> public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> "__change_events";
>
> /**
> * Local topic name for the transaction buffer snapshot.
> */
> public static final String TRANSACTION_BUFFER_SNAPSHOT =
> "__transaction_buffer_snapshot";
>
> public static final String SUBSCRIPTION_NAME =
> "transaction-buffer-sub";
> ```
> User only want to clear the backlog witch they own created and sub. But
> now we have clear the system topic sub backlog when user don't specify
> subName and clear system sub with a topic witch user created. It will bring
> many problems. etc.
> 1. clear `NAMESPACE_EVENTS_LOCAL_NAME` will clear any topic policy in this
> namespace, but user don't know and they don't want to be cleared. If the
> user does not understand the problems caused by this interface, it will
> delete many topic policy configurations and is difficult to recover, which
> will lead to unpredictable and disastrous problems.
> 2. clear `TRANSACTION_BUFFER_SNAPSHOT` topic backlog will cause
> transaction buffer recover incompletely.
> 3. clear `transaction-buffer-sub` sub also will cause transaction buffer
> recover incompletely.
> 4. if add new system topic and system sub, we should think about this
> problem.
>
> # Motivation
> We want to solve the above problems, but we can't make users unable to
> delete the backlog of system topic and system sub.
> So, we should solve types of interface
> There are two types of interfaces:
> 1. clear namespace backlog with subscriptionName
> 2. clear namespace backlog no subscriptionName
>
> when clear namespace backlog no subscriptionName:
> It represents user know the sub name is system subName and really want to
> delete it, We should not block the user clear system sub backlog and do not
> allow the user to create system sub. If the user uses it, it means that the
> user has considered the result of the clear system sub backlog
>
> when clear namespace backlog with subscriptionName:
> Because the user may not consider the existence of system topic and system
> sub, and the wrong clear backlog may lead to serious results. Therefore,
> this interface cannot delete the backlog of system topic and the backlog of
> system sub. If the user really wants to clear backlog, please bring
> subscriptionName and use another type of interface with subscriptionName.
>
> **Clear namespace and clear namespace bundle all need to handle is
> logical.**
>
> # implement
>
> ```
> class SystemTopicName {
> /**
> * Local topic name for the namespace events.
> */
> public static final String NAMESPACE_EVENTS_LOCAL_NAME =
> "__change_events";
>
> /**
> * Local topic name for the transaction buffer snapshot.
> */
> public static final String TRANSACTION_BUFFER_SNAPSHOT =
> "__transaction_buffer_snapshot";
> }
> ```
> ```
> class SystemSubName {
>
> public static final String SUBSCRIPTION_NAME =
> "transaction-buffer-sub";
>
> }
> ```
>
> 1. In order to make the code easier to maintain and reduce redundancy, we
> need to move systemTopic and systemSub to a specific classes to manage.
> 2. If the newly added systemTopic and systemSub are created from this
> class, unpredictable problems can be prevented
>
>
>
> Let me know if you have different ideas or suggestions！！
>
> Thanks!
> Bo Cong