[PIP] PIP for the Topic policy across multiple clusters

[linlin <li...@apache.org>]

Hi, All

I have prepared a PIP for the Topic policy across multiple clusters.
Please take a look and let me know what you think.

I would really appreciate it.

Best Regards,
Lin Lin

=======================================================
Topic policy across multiple clusters


   -

   Status: Proposal
   -

   Authors: Penghui Li、Chen Hang、 Lin Lin
   -

   Pull Request:
   -

   Mailing List discussion:
   -

   Release:

Motivation

When setting the topic policy for a geo-replicated cluster, some policies
want to affect the whole geo-replicated cluster but some only want to
affect the local cluster. So the proposal is to support global topic policy
and local topic policy.
Approach

Currently, we are using the TopicPolicies construction to store the topic
policy for a topic. An easy way to achieve different topic policies for
multiple clusters is to add a global flag to TopicPolicies .Replicator will
replicate policies with a global flag to other clusters

```

public class TopicPolicies {

     bBoolean isGlobal = false

}

```

We only cache one local Topic Policies in the memory before. Now it will
become two, one is Local and the other is Global.

After adding the global topic policy, the topic applied priority is:


   1.

   Local cluster topic policy
   2.

   Global topic policy
   3.

   Namespace policy
   4.

   Broker default configuration


When setting a global topic policy, we can use the `--global` option, it
should be:

```

bin/pulsar-admin topics set-retention -s 1G -t 1d --global my-topic

```

If the --global option is not added, the behavior is consistent with
before, and only updates the local policies.

Topic policies are stored in System Topic, we can directly use Replicator
to replicate data to other Clusters. We need to add a new API to the
Replicator interface, which can set the Filter Function. Then add a
Function to filter out the data with isGlobal = false

Delete a cluster?

Deleting a cluster will now delete local topic policies and will not affect
other clusters, because only global policies will be replicated to other
clusters
Changes

   -

   Every topic policy API adds the `--global` option. Including broker REST
   API, Admin SDK, CMD.



   -

   Add API `public void setFilterFunction(Function<Message, Boolean>)`. If
   Function returns false, then filter out the input message.


Compatibility

The solution does not introduce any compatibility issues, `isGlobal` in
TopicPolicies is false by default in existing Policies.

Test Plan

   1.

   Existing TopicPolicies will not be affected
   2.

   Only replicate Global Topic Policies, FilterFunction can run as expected
   3.

   Priority of Topic Policies matches our setting

Re: [PIP] PIP for the Topic policy across multiple clusters

[Sijie Guo <gu...@gmail.com>]

Hi Linlin,

Thank you for bringing this proposal to the community!

Since you are a committer, you should be able to create a wiki page on
Pulsar wiki. Can you do that?

- Sijie

On Tue, Jul 27, 2021 at 9:03 PM linlin <li...@apache.org> wrote:
>
> Hi, All
>
> I have prepared a PIP for the Topic policy across multiple clusters.
> Please take a look and let me know what you think.
>
> I would really appreciate it.
>
> Best Regards,
> Lin Lin
>
> =======================================================
> Topic policy across multiple clusters
>
>
>    -
>
>    Status: Proposal
>    -
>
>    Authors: Penghui Li、Chen Hang、 Lin Lin
>    -
>
>    Pull Request:
>    -
>
>    Mailing List discussion:
>    -
>
>    Release:
>
> Motivation
>
> When setting the topic policy for a geo-replicated cluster, some policies
> want to affect the whole geo-replicated cluster but some only want to
> affect the local cluster. So the proposal is to support global topic policy
> and local topic policy.
> Approach
>
> Currently, we are using the TopicPolicies construction to store the topic
> policy for a topic. An easy way to achieve different topic policies for
> multiple clusters is to add a global flag to TopicPolicies .Replicator will
> replicate policies with a global flag to other clusters
>
> ```
>
> public class TopicPolicies {
>
>      bBoolean isGlobal = false
>
> }
>
> ```
>
> We only cache one local Topic Policies in the memory before. Now it will
> become two, one is Local and the other is Global.
>
> After adding the global topic policy, the topic applied priority is:
>
>
>    1.
>
>    Local cluster topic policy
>    2.
>
>    Global topic policy
>    3.
>
>    Namespace policy
>    4.
>
>    Broker default configuration
>
>
> When setting a global topic policy, we can use the `--global` option, it
> should be:
>
> ```
>
> bin/pulsar-admin topics set-retention -s 1G -t 1d --global my-topic
>
> ```
>
> If the --global option is not added, the behavior is consistent with
> before, and only updates the local policies.
>
> Topic policies are stored in System Topic, we can directly use Replicator
> to replicate data to other Clusters. We need to add a new API to the
> Replicator interface, which can set the Filter Function. Then add a
> Function to filter out the data with isGlobal = false
>
> Delete a cluster?
>
> Deleting a cluster will now delete local topic policies and will not affect
> other clusters, because only global policies will be replicated to other
> clusters
> Changes
>
>    -
>
>    Every topic policy API adds the `--global` option. Including broker REST
>    API, Admin SDK, CMD.
>
>
>
>    -
>
>    Add API `public void setFilterFunction(Function<Message, Boolean>)`. If
>    Function returns false, then filter out the input message.
>
>
> Compatibility
>
> The solution does not introduce any compatibility issues, `isGlobal` in
> TopicPolicies is false by default in existing Policies.
>
> Test Plan
>
>    1.
>
>    Existing TopicPolicies will not be affected
>    2.
>
>    Only replicate Global Topic Policies, FilterFunction can run as expected
>    3.
>
>    Priority of Topic Policies matches our setting