Re: PIP: Topic policy across multiple clusters

[PengHui Li <pe...@apache.org>]

From the PulsarAdmin Java API perspective, looks like we need to add a new
param `isGlobal` for all topic policy set/get/remove methods.

Example:

setRetention(String topic, RetentionPolicies retention) ===>
setRetention(String topic, RetentionPolicies retention, boolean isGlobal);
void removeRetention(String topic) ===> void removeRetention(String topic,
boolean isGlobal);

For the get policy method, it does not make any sense to put the `applied`
and the `isGlobal` together, so looks like we need a new method
such as `getGlobalRetention(String topic)`.

But after applying the change, the Topics API looks unfriendly to users.

Another option is we can create a separate class for the topic policy such
as `pulsarAdmin.topicPolicy(boolean isGlobal)`. Since all the get applied
policy method
will not apply the `isGlobal` option, so we can create a base class
TopicPolicyBase and 2 implementations LocalTopicPolicy and
GlobalTopicPolicy.

For the usage of the Topic Policy Admin API, it should be:

```
admin.topicPolicy(true).setRetention(String topic, RetentionPolicies
retention) for setting the global retention policy for the topic,
admin.topicPolicy().setRetention(String topic, RetentionPolicies retention)
for setting the local retention policy for the topic,
admin.topicPolicy().getRetention(String topic, boolean applied) for getting
the applied policy
admin.topicPolicy().getRetention(String topic) for getting the local
retention policy
admin.topicPolicy(true).getRetention(String topic) for getting the global
retention policy
```
All the existing topic policy methods in the `admin.topics()` will not be
changed, so no compatibility issues need to be considered.
Since the `admin.topics()` have many methods which not belongs to the topic
policy, so `admin.topics(boolean isGlobal)` will confuse users.

WDYT @linlin@apache.org @Sijie Guo <si...@apache.org> @mmerli@apache.org
<mm...@apache.org>

Thanks,
Penghui

On Thu, Jul 29, 2021 at 2:37 AM linlin <fe...@gmail.com> wrote:

> Hi, All
>
> I have prepared a PIP for the Topic policy across multiple clusters.
> Please take a look and let me know what you think.
>
> I would really appreciate it.
>
> Best Regards,
> Lin Lin
>
>
>
> =======================================================================
>
> clusters
>
>
>    -
>
>    Status:
>    -
>
>    Authors: Penghui Li、Chen Hang、 Lin Lin
>    -
>
>    Pull Request:
>    -
>
>    Mailing List discussion:
>    -
>
>    Release:
>
> Motivation
>
> When setting the topic policy for a geo-replicated cluster, some policies
> want to affect the whole geo-replicated cluster but some only want to
> affect the local cluster. So the proposal is to support global topic policy
> and local topic policy.
> Approach
>
> Currently, we are using the TopicPolicies construction to store the topic
> policy for a topic. An easy way to achieve different topic policies for
> multiple clusters is to add a global flag to TopicPolicies .Replicator will
> replicate policies with a global flag to other clusters
>
> ```
>
> public class TopicPolicies {
>
>      boolean isGlobal = false
>
> }
>
> ```
>
> We only cache one local Topic Policies in the memory before. Now it will
> become two, one is Local and the other is Global.
>
> After adding the global topic policy, the topic applied priority is:
>
>
>    1.
>
>    Local cluster topic policy
>    2.
>
>    Global topic policy
>    3.
>
>    Namespace policy
>    4.
>
>    Broker default configuration
>
>
> When setting a global topic policy, we can use the `--global` option, it
> should be:
>
> ```
>
> bin/pulsar-admin topics set-retention -s 1G -t 1d --global my-topic
>
> ```
>
> If the --global option is not added, the behavior is consistent with
> before, and only updates the local policies.
>
> Topic policies are stored in System Topic, we can directly use Replicator
> to replicate data to other Clusters. We need to add a new API to the
> Replicator interface, which can set the Filter Function. Then add a
> Function to filter out the data with isGlobal = false
>
> Delete a cluster?
>
> Deleting a cluster will now delete local topic policies and will not affect
> other clusters, because only global policies will be replicated to other
> clusters
> Changes
>
>    -
>
>    Every topic policy API adds the `--global` option. Including broker REST
>    API, Admin SDK, CMD.
>
>
>
>    -
>
>    Add API `public void setFilterFunction(Function<Message, Boolean>)`. If
>    Function returns false, then filter out the input message.
>
>
> Compatibility
>
> The solution does not introduce any compatibility issues, `isGlobal` in
> TopicPolicies is false by default in existing Policies.
>
> Test Plan
>
>    1.
>
>    Existing TopicPolicies will not be affected
>    2.
>
>    Only replicate Global Topic Policies, FilterFunction can run as expected
>    3.
>
>    Priority of Topic Policies matches our setting
>