Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Michael Marshall <mm...@apache.org>]

I think this is a very appropriate direction to take Pulsar's
geo-replication. Your proposal is essentially to make the
inter-cluster configuration event driven. This increases fault
tolerance and better decouples clusters.

Thank you for your detailed proposal. After reading through it, I have
some questions :)

1. What do you think about using protobuf to define the event
protocol? I know we already have a topic policy event stream
defined with Java POJOs, but since this feature is specifically
designed for egressing cloud providers, ensuring compact data transfer
would keep egress costs down. Additionally, protobuf can help make it
clear that the schema is strict, should evolve thoughtfully, and
should be designed to work between clusters of different versions.

2. In your view, which tenant/namespace will host
`metadataSyncEventTopic`? Will there be several of these topics or is
it just hosted in a system tenant/namespace? This question gets back
to my questions about system topics on this mailing list last week [0]. I
view this topic as a system topic, so we'd need to make sure that it
has the right authorization rules and that it won't be affected by calls
like "clearNamespaceBacklog".

3. Which broker will host the metadata update publisher? I assume we
want the producer to be collocated with the bundle that hosts the
event topic. How will this be coordinated?

4. Why isn't a topic a `ResourceType`? Is this because the topic level
policies already have this feature? If so, is there a way to integrate
this feature with the existing topic policy feature?

5. By decentralizing the metadata store, it looks like there is a
chance for conflicts due to concurrent updates. How do we handle those
conflicts?

I'll also note that I previously proposed a system event topic here
[1] and it was proposed again here [2]. Those features were for
different use cases, but ultimately looked very similar. In my view, a
stream of system events is a very natural feature to expect in a
streaming technology. I wonder if there is a way to generalize this
feature to fulfill local cluster consumers and geo-replication
consumers. Even if this PIP only implements the geo-replication
portion of the feature, it'd be good to design it in an extensible fashion.

Thanks,
Michael

[0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
[1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
[2] https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx



On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <rd...@apache.org> wrote:
>
> Hi,
>
> I would like to start a discussion about PIP-136: Sync Pulsar policies
> across multiple clouds.
>
> PIP documentation: https://github.com/apache/pulsar/issues/13728
>
> *Motivation*
> Apache Pulsar is a cloud-native, distributed messaging framework which
> natively provides geo-replication. Many organizations deploy pulsar
> instances on-prem and on multiple different cloud providers and at the same
> time they would like to enable replication between multiple clusters
> deployed in different cloud providers. Pulsar already provides various
> proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to fulfill
> security requirements when brokers are deployed on different security zones
> connected with each other. However, sometimes it's not possible to share
> metadata-store (global zookeeper) between pulsar clusters deployed on
> separate cloud provider platforms, and synchronizing configuration metadata
> (policies) can be a critical path to share tenant/namespace/topic policies
> between clusters and administrate pulsar policies uniformly across all
> clusters. Therefore, we need a mechanism to sync configuration metadata
> between clusters deployed on the different cloud platforms.
>
> *Sync Pulsar policies across multiple clouds*
> https://github.com/apache/pulsar/issues/13728
> Prototype git-hub-link
> <https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6>
> Thanks,
> Rajan

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <rd...@apache.org>]

Hi,

After multiple discussions during Community meetings: I have updated PIP
with the changes and approach we agreed on. This PIP allows any
metadata-store to generate change-events using plugable
MetadataSynchronizer to publish and handle change events. PIP also uses
change-event topic as a WAL topic to persist change events and applies it
async in local and remote regions.
Can we please review the PIP: https://github.com/apache/pulsar/issues/16424
and PR: https://github.com/apache/pulsar/pull/16425

Thanks,
Rajan

On Tue, Mar 22, 2022 at 6:07 PM PengHui Li <pe...@apache.org> wrote:

> > This feature allows users to sync two separate clusters which are having
> independent global-zookeeper (metadata-store) instances. So, this feature
> will not be limited to sync a few fields of policies but think in terms of
> auto creation and syncing namespaces/tenants to entirely different
> metadata-stores where they can not talk to each other.
>
> Yes, we are on the same page. If this proposal is for syncing
> tenant/namespace/topic
> between multiple metadata stores, it looks good to me. If the proposal also
> wants to
> sync the namespace policy between multiple metadata stores, I think we
> should leverage
> the existing __change_events topic under each namespace.
>
> > Again, #12136 talks about applying policies to topics but this PIP
> addresses a separate problem where it makes it possible to integrate
> independent/isolated metadata-stores  with each other.
>
> Yes, it's not the same problem, but related. The namespace can disabled the
> geo-replication but enabled in topic level, in this case if we have
> multiple metadata store,
> and if the topic is a partitioned topic, we also need a way to sync the
> partitioned metadata
> to the remote cluster's metadata store?
>
> Thanks,
> Penghui
>
> On Wed, Mar 23, 2022 at 3:39 AM Rajan Dhabalia <rd...@apache.org>
> wrote:
>
> > >> Do we need to provide the ability for users to decide to replicate the
> > ACLs and replication cluster or not?
> >
> > This feature allows users to sync two separate clusters which are having
> > independent global-zookeeper (metadata-store) instances. So, this feature
> > will not be limited to sync a few fields of policies but think in terms
> of
> > auto creation and syncing namespaces/tenants to entirely different
> > metadata-stores where they can not talk to each other.
> >
> > >> BTW, we already supported topic level replication cluster
> > configuration[5], looks like in this case, [5]
> > https://github.com/apache/pulsar/pull/12136
> >
> > Again, #12136 talks about applying policies to topics but this PIP
> > addresses a separate problem where it makes it possible to integrate
> > independent/isolated metadata-stores  with each other.
> >
> > Thanks,
> > Rajan
> >
> > On Mon, Mar 21, 2022 at 6:29 PM PengHui Li <pe...@apache.org> wrote:
> >
> > > Thanks for the explanation.
> > >
> > > > yes, local policies doesn't need to be replicate to other clusters
> and
> > it
> > > will only replicate global policies which is shared across multiple
> > > clusters such tenant/namespace's identity-creation, ACLs, replication
> > > clusters, etc.
> > >
> > > As described in this blog [1], section "Aggregation Replication".
> > > Do we need to provide the ability for users to decide to replicate
> > > the ACLs and replication cluster or not?
> > >
> > > Currently, if users want to achieve "Aggregation Replication", it needs
> > > multiple configuration stores. So they need to maintain the namespace,
> > > partitioned topics in each cluster. A new namespace created in one
> > cluster,
> > > it need to create the namespace in other clusters if they want to
> > replicate
> > > data to those clusters.
> > >
> > > After this proposal, they don't need to create a namespace for other
> > > clusters,
> > > Pulsar will help to replicate the configuration store changes to the
> > > replicated cluster,
> > > if the new created namespace with replication cluster A, B, and C in
> > > cluster A, the
> > > namespace will be replicated to B and C.
> > >
> > > But for the ACLs and replication clusters, it should be controlled by
> > > users?
> > > e.g. only replicate the namespace to B and C, but not the replication
> > > clusters and ACLs.
> > > So that we can achieve "Aggregation Replication" with this proposal.
> > >
> > > > Topic that will be used to share policies across clusters is
> > configurable
> > > and it can be named anything. However, we should keep it a separate
> topic
> > > as it requires unique schema and special handling to synchronize
> policies
> > > across the clusters.
> > >
> > > Yes, looks like currently we already have a mechanism to replicate
> > > policies.
> > > We have a system topic under the namespace "__change_events", which
> only
> > > has
> > > topic policy changes for now. It can replicate anything under a
> > namespace.
> > > We have defined "EventType"[2] in PulsarEvent(structure used in
> > > "__change_events").
> > > And we already have a implementation for selective PulsarEvent
> > > replication[3], and schema
> > > replication[4].
> > >
> > > So it looks like we can use the "__change_events" to replicate
> namespace
> > > policies, and use a
> > > new topic which belongs to a system namespace to replicate
> > > tenant/namespace's identity-creation,
> > > partitioned topic creation?
> > >
> > > BTW, we already supported topic level replication cluster
> > configuration[5],
> > > looks like in this case,
> > > the partitioned topic is created first in one cluster without
> replication
> > > clusters first, after the replication
> > > clusters changed, pulsar will replicate the partitioned topic to remote
> > > cluster. The same mechanism is
> > > required for non-partitioned topics(users might disabled the topic
> > > auto-creation).
> > >
> > > [1]
> > >
> > >
> >
> https://www.splunk.com/en_us/blog/devops/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html
> > > [2]
> > >
> > >
> >
> https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/events/PulsarEvent.java#L36
> > > [3]
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > > [4]
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-88%3A-Replicate-schemas-across-multiple
> > > [5] https://github.com/apache/pulsar/pull/12136
> > >
> > > Regards,
> > > Penghui
> > >
> > > On Tue, Mar 22, 2022 at 6:37 AM Rajan Dhabalia <rd...@apache.org>
> > > wrote:
> > >
> > > > >> If it contains namespace policy replication, There are some
> policies
> > > no
> > > > need to replicate to another cluster
> > > > yes, local policies doesn't need to be replicate to other clusters
> and
> > it
> > > > will only replicate global policies which is shared across multiple
> > > > clusters such tenant/namespace's identity-creation, ACLs, replication
> > > > clusters, etc.
> > > >
> > > > >> The new partitioned topic also needs to be replicated to the
> remote
> > > > cluster?
> > > > Yes.
> > > >
> > > > Topic that will be used to share policies across clusters is
> > configurable
> > > > and it can be named anything. However, we should keep it a separate
> > topic
> > > > as it requires unique schema and special handling to synchronize
> > policies
> > > > across the clusters.
> > > >
> > > > Thanks,
> > > > Rajan
> > > >
> > > > On Fri, Mar 18, 2022 at 9:12 PM PengHui Li <pe...@apache.org>
> wrote:
> > > >
> > > > > Hi Rajan,
> > > > >
> > > > > Thanks for the great proposal.
> > > > >
> > > > > Will all the namespace policies be replicated to the remote
> cluster?
> > > > > I noticed the PIP title mentioned policies, but looks like from the
> > > > > `MetadataChangeEvent`,
> > > > > no namespaces policies defined. If it contains namespace policy
> > > > > replication,
> > > > > There are some policies no need to replicate to another cluster,
> > > > > for example, the rate limiter, max producers/consumers limiter.
> > > > > In
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > > > > ,
> > > > > it introduced a --global option to provide ability to apply the
> > policy
> > > in
> > > > > global or local.
> > > > >
> > > > > The new partitioned topic also needs to be replicated to the remote
> > > > > cluster?
> > > > >
> > > > > Currently, we already have a PulsarEvent struct to define the
> pulsar
> > > > system
> > > > > events,
> > > > > Looks like we can use a unified event definition by PulsarEvent.
> > > > >
> > > > > Others look good to me.
> > > > >
> > > > > Regards,
> > > > > Penghui
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com>
> wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <
> > > rdhabalia@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I would like to start VOTE on PIP-136:
> > > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rajan
> > > > > > >
> > > > > > > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <
> > > dhabalia.me@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > >
> > > > > > > > >> How do we designate the host broker? Is it manual? How
> does
> > it
> > > > > work
> > > > > > > > when the host broker is removed from the cluster?
> > > > > > > > No, it will not be manual but as I explained earlier a broker
> > > which
> > > > > > has a
> > > > > > > > failover consumer to consume remote events will be the
> > publisher
> > > > for
> > > > > > > > metadata update. If that broker is removed then a new
> failover
> > > > > > > > consumer/broker will be selected for the same.
> > > > > > > >
> > > > > > > > >> I look forward to seeing more about this design for
> conflict
> > > > > > > resolution.
> > > > > > > > Sure, I have updated PIP to handle such race condition:
> > > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > > >
> > > > > > > >
> > > > > > > > >> (1) scenarios where the Pulsar cluster operators and
> tenant
> > > > admins
> > > > > > > are
> > > > > > > > different entities and tenants can be malicious, or more
> > > probably,
> > > > > > write
> > > > > > > > bad code that will produce malicious outcomes.
> > > > > > > > I agree, Pulsar should have provision to prevent such
> scenarios
> > > > where
> > > > > > > > changes from one tenant in a cluster can impact other
> clusters.
> > > > This
> > > > > > PIP
> > > > > > > > considers the tenant/admin will be the same at both the ends
> > but
> > > > that
> > > > > > can
> > > > > > > > not be true in all cases. We can add an enhancement later or
> we
> > > can
> > > > > > > create
> > > > > > > > a separate PIP to start discussion with the possible
> solutions.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Rajan
> > > > > > > >
> > > > > > > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com>
> > > > wrote:
> > > > > > > >
> > > > > > > >> >On my first reading, it wasn't clear if there was only one
> > > topic
> > > > > > > >> required for this feature. I now see that the topic is not
> > tied
> > > > to a
> > > > > > > >> specific tenant or namespace. As such, we can avoid
> > complicated
> > > > > > > >> authorization questions by putting the required event
> topic(s)
> > > > into
> > > > > a
> > > > > > > >> "system" tenant and namespace
> > > > > > > >>
> > > > > > > >> We should consider complicated questions. We can say why we
> > > chose
> > > > > not
> > > > > > to
> > > > > > > >> address it, or why it does not apply. for a particular
> > situation
> > > > > > > >>
> > > > > > > >> Many namespace policies are administered by tenants.  As
> such
> > > any
> > > > > > tenant
> > > > > > > >> can load this topic.  Is it possible for one abusive tenant
> to
> > > > make
> > > > > > your
> > > > > > > >> system topic dysfunctional?
> > > > > > > >>
> > > > > > > >> Pulsar committers should think about
> > > > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> > > admins
> > > > > > are
> > > > > > > >> different entities and tenants can be malicious, or more
> > > probably,
> > > > > > write
> > > > > > > >> bad code that will produce malicious outcomes.
> > > > > > > >> (2) whether the changes introduce  additional SPOFs into the
> > > > > cluster.
> > > > > > > >>
> > > > > > > >> I don't think this PIP has those issues, but  as a matter of
> > > > > > practice, I
> > > > > > > >> would like to see backend/system PIPs consider these
> questions
> > > > and
> > > > > > > >> explicitly state the conclusions with rationale
> > > > > > > >>
> > > > > > > >> Joe
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <
> > > > > mmarshall@apache.org
> > > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> > Thanks for your responses.
> > > > > > > >> >
> > > > > > > >> > > I don't see a need of protobuf for this particular
> usecase
> > > > > > > >> >
> > > > > > > >> > If no one else feels strongly on this point, I am good
> with
> > > > using
> > > > > a
> > > > > > > >> POJO.
> > > > > > > >> >
> > > > > > > >> > > It doesn't matter if it's system-topic or not because
> it's
> > > > > > > >> > > configurable and the admin of the system can decide and
> > > > > configure
> > > > > > it
> > > > > > > >> > > according to the required persistent policy.
> > > > > > > >> >
> > > > > > > >> > On my first reading, it wasn't clear if there was only one
> > > topic
> > > > > > > >> > required for this feature. I now see that the topic is not
> > > tied
> > > > > to a
> > > > > > > >> > specific tenant or namespace. As such, we can avoid
> > > complicated
> > > > > > > >> > authorization questions by putting the required event
> > topic(s)
> > > > > into
> > > > > > a
> > > > > > > >> > "system" tenant and namespace, by default. The
> > `pulsar/system`
> > > > > > tenant
> > > > > > > >> > and namespace seem appropriate to me.
> > > > > > > >> >
> > > > > > > >> > > I would keep the system topic
> > > > > > > >> > > separate because this topic serves a specific purpose
> with
> > > > > > specific
> > > > > > > >> > schema,
> > > > > > > >> > > replication policy and retention policy.
> > > > > > > >> >
> > > > > > > >> > I think we need a more formal definition for system
> topics.
> > > This
> > > > > > topic
> > > > > > > >> > is exactly the kind of topic I would call a system topic:
> > its
> > > > > > intended
> > > > > > > >> > producers and consumers are Pulsar components. However,
> > > because
> > > > > > > >> > this feature can live on a topic in a system namespace, we
> > can
> > > > > avoid
> > > > > > > >> > the classification discussion for this PIP.
> > > > > > > >> >
> > > > > > > >> > > Source region will have a broker which will create a
> > > failover
> > > > > > > >> consumer on
> > > > > > > >> > > that topic and a broker with an active consumer will
> watch
> > > the
> > > > > > > >> metadata
> > > > > > > >> > > changes and publish the changes to the event topic.
> > > > > > > >> >
> > > > > > > >> > How do we designate the host broker? Is it manual? How
> does
> > it
> > > > > work
> > > > > > > >> > when the host broker is removed from the cluster?
> > > > > > > >> >
> > > > > > > >> > If we collocate the active consumer with the broker
> hosting
> > > the
> > > > > > event
> > > > > > > >> > topic, can we skip creating the failover consumer?
> > > > > > > >> >
> > > > > > > >> > > PIP briefly talks about it but I will update the PIP
> with
> > > more
> > > > > > > >> > > explanation.
> > > > > > > >> >
> > > > > > > >> > I look forward to seeing more about this design for
> conflict
> > > > > > > resolution.
> > > > > > > >> >
> > > > > > > >> > Thanks,
> > > > > > > >> > Michael
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> > > > > > dhabalia.me@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> > >
> > > > > > > >> > > Please find my response inline.
> > > > > > > >> > >
> > > > > > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > > > > > > >> mmarshall@apache.org>
> > > > > > > >> > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > > I think this is a very appropriate direction to take
> > > > Pulsar's
> > > > > > > >> > > > geo-replication. Your proposal is essentially to make
> > the
> > > > > > > >> > > > inter-cluster configuration event driven. This
> increases
> > > > fault
> > > > > > > >> > > > tolerance and better decouples clusters.
> > > > > > > >> > > >
> > > > > > > >> > > > Thank you for your detailed proposal. After reading
> > > through
> > > > > it,
> > > > > > I
> > > > > > > >> have
> > > > > > > >> > > > some questions :)
> > > > > > > >> > > >
> > > > > > > >> > > > 1. What do you think about using protobuf to define
> the
> > > > event
> > > > > > > >> > > > protocol? I know we already have a topic policy event
> > > stream
> > > > > > > >> > > > defined with Java POJOs, but since this feature is
> > > > > specifically
> > > > > > > >> > > > designed for egressing cloud providers, ensuring
> compact
> > > > data
> > > > > > > >> transfer
> > > > > > > >> > > > would keep egress costs down. Additionally, protobuf
> can
> > > > help
> > > > > > make
> > > > > > > >> it
> > > > > > > >> > > > clear that the schema is strict, should evolve
> > > thoughtfully,
> > > > > and
> > > > > > > >> > > > should be designed to work between clusters of
> different
> > > > > > versions.
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> > >  >>> I don't see a need of protobuf for this particular
> > > > usecase
> > > > > > > >> because
> > > > > > > >> > of
> > > > > > > >> > > two reasons:
> > > > > > > >> > >   >> a. policy changes don't generate huge traffic which
> > > could
> > > > > be
> > > > > > 1
> > > > > > > >> rps
> > > > > > > >> > b.
> > > > > > > >> > > and it doesn't need performance optimization.
> > > > > > > >> > >   >> It should be similar as storing policy in text
> > instead
> > > > > > protobuf
> > > > > > > >> > which
> > > > > > > >> > > doesn't impact footprint size or performance due to
> > limited
> > > > > number
> > > > > > > of
> > > > > > > >> > >  >> update operations and relatively less complexity. I
> > > agree
> > > > > that
> > > > > > > >> > protobuf
> > > > > > > >> > > could be another option but in this case it's not
> needed.
> > > > Also,
> > > > > > POJO
> > > > > > > >> > >  >> can also support schema and versioning.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > 2. In your view, which tenant/namespace will host
> > > > > > > >> > > > `metadataSyncEventTopic`? Will there be several of
> these
> > > > > topics
> > > > > > or
> > > > > > > >> is
> > > > > > > >> > > > it just hosted in a system tenant/namespace? This
> > question
> > > > > gets
> > > > > > > back
> > > > > > > >> > > > to my questions about system topics on this mailing
> list
> > > > last
> > > > > > week
> > > > > > > >> > [0]. I
> > > > > > > >> > > > view this topic as a system topic, so we'd need to
> make
> > > sure
> > > > > > that
> > > > > > > it
> > > > > > > >> > > > has the right authorization rules and that it won't be
> > > > > affected
> > > > > > by
> > > > > > > >> > calls
> > > > > > > >> > > > like "clearNamespaceBacklog".
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >   >> It doesn't matter if it's system-topic or not
> because
> > > > it's
> > > > > > > >> > > configurable and the admin of the system can decide and
> > > > > configure
> > > > > > it
> > > > > > > >> > > according to the required persistent policy. I would
> keep
> > > the
> > > > > > system
> > > > > > > >> > topic
> > > > > > > >> > > separate because this topic serves a specific purpose
> with
> > > > > > specific
> > > > > > > >> > schema,
> > > > > > > >> > > replication policy and retention policy.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > 3. Which broker will host the metadata update
> > publisher? I
> > > > > > assume
> > > > > > > we
> > > > > > > >> > > > want the producer to be collocated with the bundle
> that
> > > > hosts
> > > > > > the
> > > > > > > >> > > > event topic. How will this be coordinated?
> > > > > > > >> > > >
> > > > > > > >> > > >> It's already explained into PIP in section: "Event
> > > > publisher
> > > > > > and
> > > > > > > >> > handler"
> > > > > > > >> > > >> Every isolated cluster deployed on a separate cloud
> > > > platform
> > > > > > will
> > > > > > > >> > have a
> > > > > > > >> > > source region and part of replicated clusters for the
> > event
> > > > > topic.
> > > > > > > The
> > > > > > > >> > > Source region will have a broker which will create a
> > > failover
> > > > > > > >> consumer on
> > > > > > > >> > > that topic and a broker with an active consumer will
> watch
> > > the
> > > > > > > >> metadata
> > > > > > > >> > > changes and publish the changes to the event topic.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because
> > the
> > > > > topic
> > > > > > > >> level
> > > > > > > >> > > > policies already have this feature? If so, is there a
> > way
> > > to
> > > > > > > >> integrate
> > > > > > > >> > > > this feature with the existing topic policy feature?
> > > > > > > >> > > >
> > > > > > > >> > > >> Yes, ResourceType can be extensible to a topic as
> well.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > >
> > > > > > > >> > > > 5. By decentralizing the metadata store, it looks like
> > > there
> > > > > is
> > > > > > a
> > > > > > > >> > > > chance for conflicts due to concurrent updates. How do
> > we
> > > > > handle
> > > > > > > >> those
> > > > > > > >> > > > conflicts?
> > > > > > > >> > > >
> > > > > > > >> > > >>  PIP briefly talks about it but I will update the PIP
> > > with
> > > > > more
> > > > > > > >> > > explanation. MetadataChangeEvent contains source-cluster
> > and
> > > > > > updated
> > > > > > > >> > time.
> > > > > > > >> > > Also, resources Tenant/Namespace will also contain
> > > > > lastUpdatedTime
> > > > > > > >> which
> > > > > > > >> > > will help to destination clusters to handle
> > stale/duplicate
> > > > > events
> > > > > > > and
> > > > > > > >> > race
> > > > > > > >> > > conditions. Also, snapshot-sync an additional task helps
> > all
> > > > > > > clusters
> > > > > > > >> to
> > > > > > > >> > be
> > > > > > > >> > > synced with each other eventually.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > > I'll also note that I previously proposed a system
> event
> > > > topic
> > > > > > > here
> > > > > > > >> > > > [1] and it was proposed again here [2]. Those features
> > > were
> > > > > for
> > > > > > > >> > > > different use cases, but ultimately looked very
> similar.
> > > In
> > > > my
> > > > > > > >> view, a
> > > > > > > >> > > > stream of system events is a very natural feature to
> > > expect
> > > > > in a
> > > > > > > >> > > > streaming technology. I wonder if there is a way to
> > > > generalize
> > > > > > > this
> > > > > > > >> > > > feature to fulfill local cluster consumers and
> > > > geo-replication
> > > > > > > >> > > > consumers. Even if this PIP only implements the
> > > > > geo-replication
> > > > > > > >> > > > portion of the feature, it'd be good to design it in
> an
> > > > > > extensible
> > > > > > > >> > fashion.
> > > > > > > >> > > >
> > > > > > > >> > >  >> I think answer (2) addresses this concern as well.
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > > Thanks,
> > > > > > > >> > > > Michael
> > > > > > > >> > > >
> > > > > > > >> > > > [0]
> > > > > > > >>
> > > https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > > > > >> > > > [1]
> > > > > > > >>
> > > https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > > > > > >> > > > [2]
> > > > > > > >>
> > > https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > >
> > > > > > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > > > > > > >> rdhabalia@apache.org>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > Hi,
> > > > > > > >> > > > >
> > > > > > > >> > > > > I would like to start a discussion about PIP-136:
> Sync
> > > > > Pulsar
> > > > > > > >> > policies
> > > > > > > >> > > > > across multiple clouds.
> > > > > > > >> > > > >
> > > > > > > >> > > > > PIP documentation:
> > > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > > >> > > > >
> > > > > > > >> > > > > *Motivation*
> > > > > > > >> > > > > Apache Pulsar is a cloud-native, distributed
> messaging
> > > > > > framework
> > > > > > > >> > which
> > > > > > > >> > > > > natively provides geo-replication. Many
> organizations
> > > > deploy
> > > > > > > >> pulsar
> > > > > > > >> > > > > instances on-prem and on multiple different cloud
> > > > providers
> > > > > > and
> > > > > > > at
> > > > > > > >> > the
> > > > > > > >> > > > same
> > > > > > > >> > > > > time they would like to enable replication between
> > > > multiple
> > > > > > > >> clusters
> > > > > > > >> > > > > deployed in different cloud providers. Pulsar
> already
> > > > > provides
> > > > > > > >> > various
> > > > > > > >> > > > > proxy options (Pulsar proxy/ enterprise proxy
> > solutions
> > > on
> > > > > > SNI)
> > > > > > > to
> > > > > > > >> > > > fulfill
> > > > > > > >> > > > > security requirements when brokers are deployed on
> > > > different
> > > > > > > >> security
> > > > > > > >> > > > zones
> > > > > > > >> > > > > connected with each other. However, sometimes it's
> not
> > > > > > possible
> > > > > > > to
> > > > > > > >> > share
> > > > > > > >> > > > > metadata-store (global zookeeper) between pulsar
> > > clusters
> > > > > > > >> deployed on
> > > > > > > >> > > > > separate cloud provider platforms, and synchronizing
> > > > > > > configuration
> > > > > > > >> > > > metadata
> > > > > > > >> > > > > (policies) can be a critical path to share
> > > > > > > tenant/namespace/topic
> > > > > > > >> > > > policies
> > > > > > > >> > > > > between clusters and administrate pulsar policies
> > > > uniformly
> > > > > > > across
> > > > > > > >> > all
> > > > > > > >> > > > > clusters. Therefore, we need a mechanism to sync
> > > > > configuration
> > > > > > > >> > metadata
> > > > > > > >> > > > > between clusters deployed on the different cloud
> > > > platforms.
> > > > > > > >> > > > >
> > > > > > > >> > > > > *Sync Pulsar policies across multiple clouds*
> > > > > > > >> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > > >> > > > > Prototype git-hub-link
> > > > > > > >> > > > > <
> > > > > > > >> > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks,
> > > > > > > >> > > > > Rajan
> > > > > > > >> > > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[PengHui Li <pe...@apache.org>]

> This feature allows users to sync two separate clusters which are having
independent global-zookeeper (metadata-store) instances. So, this feature
will not be limited to sync a few fields of policies but think in terms of
auto creation and syncing namespaces/tenants to entirely different
metadata-stores where they can not talk to each other.

Yes, we are on the same page. If this proposal is for syncing
tenant/namespace/topic
between multiple metadata stores, it looks good to me. If the proposal also
wants to
sync the namespace policy between multiple metadata stores, I think we
should leverage
the existing __change_events topic under each namespace.

> Again, #12136 talks about applying policies to topics but this PIP
addresses a separate problem where it makes it possible to integrate
independent/isolated metadata-stores  with each other.

Yes, it's not the same problem, but related. The namespace can disabled the
geo-replication but enabled in topic level, in this case if we have
multiple metadata store,
and if the topic is a partitioned topic, we also need a way to sync the
partitioned metadata
to the remote cluster's metadata store?

Thanks,
Penghui

On Wed, Mar 23, 2022 at 3:39 AM Rajan Dhabalia <rd...@apache.org> wrote:

> >> Do we need to provide the ability for users to decide to replicate the
> ACLs and replication cluster or not?
>
> This feature allows users to sync two separate clusters which are having
> independent global-zookeeper (metadata-store) instances. So, this feature
> will not be limited to sync a few fields of policies but think in terms of
> auto creation and syncing namespaces/tenants to entirely different
> metadata-stores where they can not talk to each other.
>
> >> BTW, we already supported topic level replication cluster
> configuration[5], looks like in this case, [5]
> https://github.com/apache/pulsar/pull/12136
>
> Again, #12136 talks about applying policies to topics but this PIP
> addresses a separate problem where it makes it possible to integrate
> independent/isolated metadata-stores  with each other.
>
> Thanks,
> Rajan
>
> On Mon, Mar 21, 2022 at 6:29 PM PengHui Li <pe...@apache.org> wrote:
>
> > Thanks for the explanation.
> >
> > > yes, local policies doesn't need to be replicate to other clusters and
> it
> > will only replicate global policies which is shared across multiple
> > clusters such tenant/namespace's identity-creation, ACLs, replication
> > clusters, etc.
> >
> > As described in this blog [1], section "Aggregation Replication".
> > Do we need to provide the ability for users to decide to replicate
> > the ACLs and replication cluster or not?
> >
> > Currently, if users want to achieve "Aggregation Replication", it needs
> > multiple configuration stores. So they need to maintain the namespace,
> > partitioned topics in each cluster. A new namespace created in one
> cluster,
> > it need to create the namespace in other clusters if they want to
> replicate
> > data to those clusters.
> >
> > After this proposal, they don't need to create a namespace for other
> > clusters,
> > Pulsar will help to replicate the configuration store changes to the
> > replicated cluster,
> > if the new created namespace with replication cluster A, B, and C in
> > cluster A, the
> > namespace will be replicated to B and C.
> >
> > But for the ACLs and replication clusters, it should be controlled by
> > users?
> > e.g. only replicate the namespace to B and C, but not the replication
> > clusters and ACLs.
> > So that we can achieve "Aggregation Replication" with this proposal.
> >
> > > Topic that will be used to share policies across clusters is
> configurable
> > and it can be named anything. However, we should keep it a separate topic
> > as it requires unique schema and special handling to synchronize policies
> > across the clusters.
> >
> > Yes, looks like currently we already have a mechanism to replicate
> > policies.
> > We have a system topic under the namespace "__change_events", which only
> > has
> > topic policy changes for now. It can replicate anything under a
> namespace.
> > We have defined "EventType"[2] in PulsarEvent(structure used in
> > "__change_events").
> > And we already have a implementation for selective PulsarEvent
> > replication[3], and schema
> > replication[4].
> >
> > So it looks like we can use the "__change_events" to replicate namespace
> > policies, and use a
> > new topic which belongs to a system namespace to replicate
> > tenant/namespace's identity-creation,
> > partitioned topic creation?
> >
> > BTW, we already supported topic level replication cluster
> configuration[5],
> > looks like in this case,
> > the partitioned topic is created first in one cluster without replication
> > clusters first, after the replication
> > clusters changed, pulsar will replicate the partitioned topic to remote
> > cluster. The same mechanism is
> > required for non-partitioned topics(users might disabled the topic
> > auto-creation).
> >
> > [1]
> >
> >
> https://www.splunk.com/en_us/blog/devops/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html
> > [2]
> >
> >
> https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/events/PulsarEvent.java#L36
> > [3]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > [4]
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-88%3A-Replicate-schemas-across-multiple
> > [5] https://github.com/apache/pulsar/pull/12136
> >
> > Regards,
> > Penghui
> >
> > On Tue, Mar 22, 2022 at 6:37 AM Rajan Dhabalia <rd...@apache.org>
> > wrote:
> >
> > > >> If it contains namespace policy replication, There are some policies
> > no
> > > need to replicate to another cluster
> > > yes, local policies doesn't need to be replicate to other clusters and
> it
> > > will only replicate global policies which is shared across multiple
> > > clusters such tenant/namespace's identity-creation, ACLs, replication
> > > clusters, etc.
> > >
> > > >> The new partitioned topic also needs to be replicated to the remote
> > > cluster?
> > > Yes.
> > >
> > > Topic that will be used to share policies across clusters is
> configurable
> > > and it can be named anything. However, we should keep it a separate
> topic
> > > as it requires unique schema and special handling to synchronize
> policies
> > > across the clusters.
> > >
> > > Thanks,
> > > Rajan
> > >
> > > On Fri, Mar 18, 2022 at 9:12 PM PengHui Li <pe...@apache.org> wrote:
> > >
> > > > Hi Rajan,
> > > >
> > > > Thanks for the great proposal.
> > > >
> > > > Will all the namespace policies be replicated to the remote cluster?
> > > > I noticed the PIP title mentioned policies, but looks like from the
> > > > `MetadataChangeEvent`,
> > > > no namespaces policies defined. If it contains namespace policy
> > > > replication,
> > > > There are some policies no need to replicate to another cluster,
> > > > for example, the rate limiter, max producers/consumers limiter.
> > > > In
> > > >
> > > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > > > ,
> > > > it introduced a --global option to provide ability to apply the
> policy
> > in
> > > > global or local.
> > > >
> > > > The new partitioned topic also needs to be replicated to the remote
> > > > cluster?
> > > >
> > > > Currently, we already have a PulsarEvent struct to define the pulsar
> > > system
> > > > events,
> > > > Looks like we can use a unified event definition by PulsarEvent.
> > > >
> > > > Others look good to me.
> > > >
> > > > Regards,
> > > > Penghui
> > > >
> > > >
> > > >
> > > > On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <
> > rdhabalia@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I would like to start VOTE on PIP-136:
> > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > >
> > > > > > Thanks,
> > > > > > Rajan
> > > > > >
> > > > > > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <
> > dhabalia.me@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > >
> > > > > > > >> How do we designate the host broker? Is it manual? How does
> it
> > > > work
> > > > > > > when the host broker is removed from the cluster?
> > > > > > > No, it will not be manual but as I explained earlier a broker
> > which
> > > > > has a
> > > > > > > failover consumer to consume remote events will be the
> publisher
> > > for
> > > > > > > metadata update. If that broker is removed then a new failover
> > > > > > > consumer/broker will be selected for the same.
> > > > > > >
> > > > > > > >> I look forward to seeing more about this design for conflict
> > > > > > resolution.
> > > > > > > Sure, I have updated PIP to handle such race condition:
> > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > >
> > > > > > >
> > > > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> > > admins
> > > > > > are
> > > > > > > different entities and tenants can be malicious, or more
> > probably,
> > > > > write
> > > > > > > bad code that will produce malicious outcomes.
> > > > > > > I agree, Pulsar should have provision to prevent such scenarios
> > > where
> > > > > > > changes from one tenant in a cluster can impact other clusters.
> > > This
> > > > > PIP
> > > > > > > considers the tenant/admin will be the same at both the ends
> but
> > > that
> > > > > can
> > > > > > > not be true in all cases. We can add an enhancement later or we
> > can
> > > > > > create
> > > > > > > a separate PIP to start discussion with the possible solutions.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Rajan
> > > > > > >
> > > > > > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com>
> > > wrote:
> > > > > > >
> > > > > > >> >On my first reading, it wasn't clear if there was only one
> > topic
> > > > > > >> required for this feature. I now see that the topic is not
> tied
> > > to a
> > > > > > >> specific tenant or namespace. As such, we can avoid
> complicated
> > > > > > >> authorization questions by putting the required event topic(s)
> > > into
> > > > a
> > > > > > >> "system" tenant and namespace
> > > > > > >>
> > > > > > >> We should consider complicated questions. We can say why we
> > chose
> > > > not
> > > > > to
> > > > > > >> address it, or why it does not apply. for a particular
> situation
> > > > > > >>
> > > > > > >> Many namespace policies are administered by tenants.  As such
> > any
> > > > > tenant
> > > > > > >> can load this topic.  Is it possible for one abusive tenant to
> > > make
> > > > > your
> > > > > > >> system topic dysfunctional?
> > > > > > >>
> > > > > > >> Pulsar committers should think about
> > > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> > admins
> > > > > are
> > > > > > >> different entities and tenants can be malicious, or more
> > probably,
> > > > > write
> > > > > > >> bad code that will produce malicious outcomes.
> > > > > > >> (2) whether the changes introduce  additional SPOFs into the
> > > > cluster.
> > > > > > >>
> > > > > > >> I don't think this PIP has those issues, but  as a matter of
> > > > > practice, I
> > > > > > >> would like to see backend/system PIPs consider these questions
> > > and
> > > > > > >> explicitly state the conclusions with rationale
> > > > > > >>
> > > > > > >> Joe
> > > > > > >>
> > > > > > >>
> > > > > > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <
> > > > mmarshall@apache.org
> > > > > >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Thanks for your responses.
> > > > > > >> >
> > > > > > >> > > I don't see a need of protobuf for this particular usecase
> > > > > > >> >
> > > > > > >> > If no one else feels strongly on this point, I am good with
> > > using
> > > > a
> > > > > > >> POJO.
> > > > > > >> >
> > > > > > >> > > It doesn't matter if it's system-topic or not because it's
> > > > > > >> > > configurable and the admin of the system can decide and
> > > > configure
> > > > > it
> > > > > > >> > > according to the required persistent policy.
> > > > > > >> >
> > > > > > >> > On my first reading, it wasn't clear if there was only one
> > topic
> > > > > > >> > required for this feature. I now see that the topic is not
> > tied
> > > > to a
> > > > > > >> > specific tenant or namespace. As such, we can avoid
> > complicated
> > > > > > >> > authorization questions by putting the required event
> topic(s)
> > > > into
> > > > > a
> > > > > > >> > "system" tenant and namespace, by default. The
> `pulsar/system`
> > > > > tenant
> > > > > > >> > and namespace seem appropriate to me.
> > > > > > >> >
> > > > > > >> > > I would keep the system topic
> > > > > > >> > > separate because this topic serves a specific purpose with
> > > > > specific
> > > > > > >> > schema,
> > > > > > >> > > replication policy and retention policy.
> > > > > > >> >
> > > > > > >> > I think we need a more formal definition for system topics.
> > This
> > > > > topic
> > > > > > >> > is exactly the kind of topic I would call a system topic:
> its
> > > > > intended
> > > > > > >> > producers and consumers are Pulsar components. However,
> > because
> > > > > > >> > this feature can live on a topic in a system namespace, we
> can
> > > > avoid
> > > > > > >> > the classification discussion for this PIP.
> > > > > > >> >
> > > > > > >> > > Source region will have a broker which will create a
> > failover
> > > > > > >> consumer on
> > > > > > >> > > that topic and a broker with an active consumer will watch
> > the
> > > > > > >> metadata
> > > > > > >> > > changes and publish the changes to the event topic.
> > > > > > >> >
> > > > > > >> > How do we designate the host broker? Is it manual? How does
> it
> > > > work
> > > > > > >> > when the host broker is removed from the cluster?
> > > > > > >> >
> > > > > > >> > If we collocate the active consumer with the broker hosting
> > the
> > > > > event
> > > > > > >> > topic, can we skip creating the failover consumer?
> > > > > > >> >
> > > > > > >> > > PIP briefly talks about it but I will update the PIP with
> > more
> > > > > > >> > > explanation.
> > > > > > >> >
> > > > > > >> > I look forward to seeing more about this design for conflict
> > > > > > resolution.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Michael
> > > > > > >> >
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> > > > > dhabalia.me@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> > >
> > > > > > >> > > Please find my response inline.
> > > > > > >> > >
> > > > > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > > > > > >> mmarshall@apache.org>
> > > > > > >> > > wrote:
> > > > > > >> > >
> > > > > > >> > > > I think this is a very appropriate direction to take
> > > Pulsar's
> > > > > > >> > > > geo-replication. Your proposal is essentially to make
> the
> > > > > > >> > > > inter-cluster configuration event driven. This increases
> > > fault
> > > > > > >> > > > tolerance and better decouples clusters.
> > > > > > >> > > >
> > > > > > >> > > > Thank you for your detailed proposal. After reading
> > through
> > > > it,
> > > > > I
> > > > > > >> have
> > > > > > >> > > > some questions :)
> > > > > > >> > > >
> > > > > > >> > > > 1. What do you think about using protobuf to define the
> > > event
> > > > > > >> > > > protocol? I know we already have a topic policy event
> > stream
> > > > > > >> > > > defined with Java POJOs, but since this feature is
> > > > specifically
> > > > > > >> > > > designed for egressing cloud providers, ensuring compact
> > > data
> > > > > > >> transfer
> > > > > > >> > > > would keep egress costs down. Additionally, protobuf can
> > > help
> > > > > make
> > > > > > >> it
> > > > > > >> > > > clear that the schema is strict, should evolve
> > thoughtfully,
> > > > and
> > > > > > >> > > > should be designed to work between clusters of different
> > > > > versions.
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> > >  >>> I don't see a need of protobuf for this particular
> > > usecase
> > > > > > >> because
> > > > > > >> > of
> > > > > > >> > > two reasons:
> > > > > > >> > >   >> a. policy changes don't generate huge traffic which
> > could
> > > > be
> > > > > 1
> > > > > > >> rps
> > > > > > >> > b.
> > > > > > >> > > and it doesn't need performance optimization.
> > > > > > >> > >   >> It should be similar as storing policy in text
> instead
> > > > > protobuf
> > > > > > >> > which
> > > > > > >> > > doesn't impact footprint size or performance due to
> limited
> > > > number
> > > > > > of
> > > > > > >> > >  >> update operations and relatively less complexity. I
> > agree
> > > > that
> > > > > > >> > protobuf
> > > > > > >> > > could be another option but in this case it's not needed.
> > > Also,
> > > > > POJO
> > > > > > >> > >  >> can also support schema and versioning.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > 2. In your view, which tenant/namespace will host
> > > > > > >> > > > `metadataSyncEventTopic`? Will there be several of these
> > > > topics
> > > > > or
> > > > > > >> is
> > > > > > >> > > > it just hosted in a system tenant/namespace? This
> question
> > > > gets
> > > > > > back
> > > > > > >> > > > to my questions about system topics on this mailing list
> > > last
> > > > > week
> > > > > > >> > [0]. I
> > > > > > >> > > > view this topic as a system topic, so we'd need to make
> > sure
> > > > > that
> > > > > > it
> > > > > > >> > > > has the right authorization rules and that it won't be
> > > > affected
> > > > > by
> > > > > > >> > calls
> > > > > > >> > > > like "clearNamespaceBacklog".
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >   >> It doesn't matter if it's system-topic or not because
> > > it's
> > > > > > >> > > configurable and the admin of the system can decide and
> > > > configure
> > > > > it
> > > > > > >> > > according to the required persistent policy. I would keep
> > the
> > > > > system
> > > > > > >> > topic
> > > > > > >> > > separate because this topic serves a specific purpose with
> > > > > specific
> > > > > > >> > schema,
> > > > > > >> > > replication policy and retention policy.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > 3. Which broker will host the metadata update
> publisher? I
> > > > > assume
> > > > > > we
> > > > > > >> > > > want the producer to be collocated with the bundle that
> > > hosts
> > > > > the
> > > > > > >> > > > event topic. How will this be coordinated?
> > > > > > >> > > >
> > > > > > >> > > >> It's already explained into PIP in section: "Event
> > > publisher
> > > > > and
> > > > > > >> > handler"
> > > > > > >> > > >> Every isolated cluster deployed on a separate cloud
> > > platform
> > > > > will
> > > > > > >> > have a
> > > > > > >> > > source region and part of replicated clusters for the
> event
> > > > topic.
> > > > > > The
> > > > > > >> > > Source region will have a broker which will create a
> > failover
> > > > > > >> consumer on
> > > > > > >> > > that topic and a broker with an active consumer will watch
> > the
> > > > > > >> metadata
> > > > > > >> > > changes and publish the changes to the event topic.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because
> the
> > > > topic
> > > > > > >> level
> > > > > > >> > > > policies already have this feature? If so, is there a
> way
> > to
> > > > > > >> integrate
> > > > > > >> > > > this feature with the existing topic policy feature?
> > > > > > >> > > >
> > > > > > >> > > >> Yes, ResourceType can be extensible to a topic as well.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > >
> > > > > > >> > > > 5. By decentralizing the metadata store, it looks like
> > there
> > > > is
> > > > > a
> > > > > > >> > > > chance for conflicts due to concurrent updates. How do
> we
> > > > handle
> > > > > > >> those
> > > > > > >> > > > conflicts?
> > > > > > >> > > >
> > > > > > >> > > >>  PIP briefly talks about it but I will update the PIP
> > with
> > > > more
> > > > > > >> > > explanation. MetadataChangeEvent contains source-cluster
> and
> > > > > updated
> > > > > > >> > time.
> > > > > > >> > > Also, resources Tenant/Namespace will also contain
> > > > lastUpdatedTime
> > > > > > >> which
> > > > > > >> > > will help to destination clusters to handle
> stale/duplicate
> > > > events
> > > > > > and
> > > > > > >> > race
> > > > > > >> > > conditions. Also, snapshot-sync an additional task helps
> all
> > > > > > clusters
> > > > > > >> to
> > > > > > >> > be
> > > > > > >> > > synced with each other eventually.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > > I'll also note that I previously proposed a system event
> > > topic
> > > > > > here
> > > > > > >> > > > [1] and it was proposed again here [2]. Those features
> > were
> > > > for
> > > > > > >> > > > different use cases, but ultimately looked very similar.
> > In
> > > my
> > > > > > >> view, a
> > > > > > >> > > > stream of system events is a very natural feature to
> > expect
> > > > in a
> > > > > > >> > > > streaming technology. I wonder if there is a way to
> > > generalize
> > > > > > this
> > > > > > >> > > > feature to fulfill local cluster consumers and
> > > geo-replication
> > > > > > >> > > > consumers. Even if this PIP only implements the
> > > > geo-replication
> > > > > > >> > > > portion of the feature, it'd be good to design it in an
> > > > > extensible
> > > > > > >> > fashion.
> > > > > > >> > > >
> > > > > > >> > >  >> I think answer (2) addresses this concern as well.
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > > Thanks,
> > > > > > >> > > > Michael
> > > > > > >> > > >
> > > > > > >> > > > [0]
> > > > > > >>
> > https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > > > >> > > > [1]
> > > > > > >>
> > https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > > > > >> > > > [2]
> > > > > > >>
> > https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > >
> > > > > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > > > > > >> rdhabalia@apache.org>
> > > > > > >> > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > Hi,
> > > > > > >> > > > >
> > > > > > >> > > > > I would like to start a discussion about PIP-136: Sync
> > > > Pulsar
> > > > > > >> > policies
> > > > > > >> > > > > across multiple clouds.
> > > > > > >> > > > >
> > > > > > >> > > > > PIP documentation:
> > > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > >> > > > >
> > > > > > >> > > > > *Motivation*
> > > > > > >> > > > > Apache Pulsar is a cloud-native, distributed messaging
> > > > > framework
> > > > > > >> > which
> > > > > > >> > > > > natively provides geo-replication. Many organizations
> > > deploy
> > > > > > >> pulsar
> > > > > > >> > > > > instances on-prem and on multiple different cloud
> > > providers
> > > > > and
> > > > > > at
> > > > > > >> > the
> > > > > > >> > > > same
> > > > > > >> > > > > time they would like to enable replication between
> > > multiple
> > > > > > >> clusters
> > > > > > >> > > > > deployed in different cloud providers. Pulsar already
> > > > provides
> > > > > > >> > various
> > > > > > >> > > > > proxy options (Pulsar proxy/ enterprise proxy
> solutions
> > on
> > > > > SNI)
> > > > > > to
> > > > > > >> > > > fulfill
> > > > > > >> > > > > security requirements when brokers are deployed on
> > > different
> > > > > > >> security
> > > > > > >> > > > zones
> > > > > > >> > > > > connected with each other. However, sometimes it's not
> > > > > possible
> > > > > > to
> > > > > > >> > share
> > > > > > >> > > > > metadata-store (global zookeeper) between pulsar
> > clusters
> > > > > > >> deployed on
> > > > > > >> > > > > separate cloud provider platforms, and synchronizing
> > > > > > configuration
> > > > > > >> > > > metadata
> > > > > > >> > > > > (policies) can be a critical path to share
> > > > > > tenant/namespace/topic
> > > > > > >> > > > policies
> > > > > > >> > > > > between clusters and administrate pulsar policies
> > > uniformly
> > > > > > across
> > > > > > >> > all
> > > > > > >> > > > > clusters. Therefore, we need a mechanism to sync
> > > > configuration
> > > > > > >> > metadata
> > > > > > >> > > > > between clusters deployed on the different cloud
> > > platforms.
> > > > > > >> > > > >
> > > > > > >> > > > > *Sync Pulsar policies across multiple clouds*
> > > > > > >> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > > >> > > > > Prototype git-hub-link
> > > > > > >> > > > > <
> > > > > > >> > > >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks,
> > > > > > >> > > > > Rajan
> > > > > > >> > > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <rd...@apache.org>]

>> Do we need to provide the ability for users to decide to replicate the
ACLs and replication cluster or not?

This feature allows users to sync two separate clusters which are having
independent global-zookeeper (metadata-store) instances. So, this feature
will not be limited to sync a few fields of policies but think in terms of
auto creation and syncing namespaces/tenants to entirely different
metadata-stores where they can not talk to each other.

>> BTW, we already supported topic level replication cluster
configuration[5], looks like in this case, [5]
https://github.com/apache/pulsar/pull/12136

Again, #12136 talks about applying policies to topics but this PIP
addresses a separate problem where it makes it possible to integrate
independent/isolated metadata-stores  with each other.

Thanks,
Rajan

On Mon, Mar 21, 2022 at 6:29 PM PengHui Li <pe...@apache.org> wrote:

> Thanks for the explanation.
>
> > yes, local policies doesn't need to be replicate to other clusters and it
> will only replicate global policies which is shared across multiple
> clusters such tenant/namespace's identity-creation, ACLs, replication
> clusters, etc.
>
> As described in this blog [1], section "Aggregation Replication".
> Do we need to provide the ability for users to decide to replicate
> the ACLs and replication cluster or not?
>
> Currently, if users want to achieve "Aggregation Replication", it needs
> multiple configuration stores. So they need to maintain the namespace,
> partitioned topics in each cluster. A new namespace created in one cluster,
> it need to create the namespace in other clusters if they want to replicate
> data to those clusters.
>
> After this proposal, they don't need to create a namespace for other
> clusters,
> Pulsar will help to replicate the configuration store changes to the
> replicated cluster,
> if the new created namespace with replication cluster A, B, and C in
> cluster A, the
> namespace will be replicated to B and C.
>
> But for the ACLs and replication clusters, it should be controlled by
> users?
> e.g. only replicate the namespace to B and C, but not the replication
> clusters and ACLs.
> So that we can achieve "Aggregation Replication" with this proposal.
>
> > Topic that will be used to share policies across clusters is configurable
> and it can be named anything. However, we should keep it a separate topic
> as it requires unique schema and special handling to synchronize policies
> across the clusters.
>
> Yes, looks like currently we already have a mechanism to replicate
> policies.
> We have a system topic under the namespace "__change_events", which only
> has
> topic policy changes for now. It can replicate anything under a namespace.
> We have defined "EventType"[2] in PulsarEvent(structure used in
> "__change_events").
> And we already have a implementation for selective PulsarEvent
> replication[3], and schema
> replication[4].
>
> So it looks like we can use the "__change_events" to replicate namespace
> policies, and use a
> new topic which belongs to a system namespace to replicate
> tenant/namespace's identity-creation,
> partitioned topic creation?
>
> BTW, we already supported topic level replication cluster configuration[5],
> looks like in this case,
> the partitioned topic is created first in one cluster without replication
> clusters first, after the replication
> clusters changed, pulsar will replicate the partitioned topic to remote
> cluster. The same mechanism is
> required for non-partitioned topics(users might disabled the topic
> auto-creation).
>
> [1]
>
> https://www.splunk.com/en_us/blog/devops/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html
> [2]
>
> https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/events/PulsarEvent.java#L36
> [3]
>
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> [4]
>
> https://github.com/apache/pulsar/wiki/PIP-88%3A-Replicate-schemas-across-multiple
> [5] https://github.com/apache/pulsar/pull/12136
>
> Regards,
> Penghui
>
> On Tue, Mar 22, 2022 at 6:37 AM Rajan Dhabalia <rd...@apache.org>
> wrote:
>
> > >> If it contains namespace policy replication, There are some policies
> no
> > need to replicate to another cluster
> > yes, local policies doesn't need to be replicate to other clusters and it
> > will only replicate global policies which is shared across multiple
> > clusters such tenant/namespace's identity-creation, ACLs, replication
> > clusters, etc.
> >
> > >> The new partitioned topic also needs to be replicated to the remote
> > cluster?
> > Yes.
> >
> > Topic that will be used to share policies across clusters is configurable
> > and it can be named anything. However, we should keep it a separate topic
> > as it requires unique schema and special handling to synchronize policies
> > across the clusters.
> >
> > Thanks,
> > Rajan
> >
> > On Fri, Mar 18, 2022 at 9:12 PM PengHui Li <pe...@apache.org> wrote:
> >
> > > Hi Rajan,
> > >
> > > Thanks for the great proposal.
> > >
> > > Will all the namespace policies be replicated to the remote cluster?
> > > I noticed the PIP title mentioned policies, but looks like from the
> > > `MetadataChangeEvent`,
> > > no namespaces policies defined. If it contains namespace policy
> > > replication,
> > > There are some policies no need to replicate to another cluster,
> > > for example, the rate limiter, max producers/consumers limiter.
> > > In
> > >
> > >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > > ,
> > > it introduced a --global option to provide ability to apply the policy
> in
> > > global or local.
> > >
> > > The new partitioned topic also needs to be replicated to the remote
> > > cluster?
> > >
> > > Currently, we already have a PulsarEvent struct to define the pulsar
> > system
> > > events,
> > > Looks like we can use a unified event definition by PulsarEvent.
> > >
> > > Others look good to me.
> > >
> > > Regards,
> > > Penghui
> > >
> > >
> > >
> > > On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com> wrote:
> > >
> > > > +1
> > > >
> > > > On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <
> rdhabalia@apache.org>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I would like to start VOTE on PIP-136:
> > > > > https://github.com/apache/pulsar/issues/13728
> > > > >
> > > > > Thanks,
> > > > > Rajan
> > > > >
> > > > > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <
> dhabalia.me@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > >
> > > > > > >> How do we designate the host broker? Is it manual? How does it
> > > work
> > > > > > when the host broker is removed from the cluster?
> > > > > > No, it will not be manual but as I explained earlier a broker
> which
> > > > has a
> > > > > > failover consumer to consume remote events will be the publisher
> > for
> > > > > > metadata update. If that broker is removed then a new failover
> > > > > > consumer/broker will be selected for the same.
> > > > > >
> > > > > > >> I look forward to seeing more about this design for conflict
> > > > > resolution.
> > > > > > Sure, I have updated PIP to handle such race condition:
> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > >
> > > > > >
> > > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> > admins
> > > > > are
> > > > > > different entities and tenants can be malicious, or more
> probably,
> > > > write
> > > > > > bad code that will produce malicious outcomes.
> > > > > > I agree, Pulsar should have provision to prevent such scenarios
> > where
> > > > > > changes from one tenant in a cluster can impact other clusters.
> > This
> > > > PIP
> > > > > > considers the tenant/admin will be the same at both the ends but
> > that
> > > > can
> > > > > > not be true in all cases. We can add an enhancement later or we
> can
> > > > > create
> > > > > > a separate PIP to start discussion with the possible solutions.
> > > > > >
> > > > > > Thanks,
> > > > > > Rajan
> > > > > >
> > > > > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com>
> > wrote:
> > > > > >
> > > > > >> >On my first reading, it wasn't clear if there was only one
> topic
> > > > > >> required for this feature. I now see that the topic is not tied
> > to a
> > > > > >> specific tenant or namespace. As such, we can avoid complicated
> > > > > >> authorization questions by putting the required event topic(s)
> > into
> > > a
> > > > > >> "system" tenant and namespace
> > > > > >>
> > > > > >> We should consider complicated questions. We can say why we
> chose
> > > not
> > > > to
> > > > > >> address it, or why it does not apply. for a particular situation
> > > > > >>
> > > > > >> Many namespace policies are administered by tenants.  As such
> any
> > > > tenant
> > > > > >> can load this topic.  Is it possible for one abusive tenant to
> > make
> > > > your
> > > > > >> system topic dysfunctional?
> > > > > >>
> > > > > >> Pulsar committers should think about
> > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> admins
> > > > are
> > > > > >> different entities and tenants can be malicious, or more
> probably,
> > > > write
> > > > > >> bad code that will produce malicious outcomes.
> > > > > >> (2) whether the changes introduce  additional SPOFs into the
> > > cluster.
> > > > > >>
> > > > > >> I don't think this PIP has those issues, but  as a matter of
> > > > practice, I
> > > > > >> would like to see backend/system PIPs consider these questions
> > and
> > > > > >> explicitly state the conclusions with rationale
> > > > > >>
> > > > > >> Joe
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <
> > > mmarshall@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Thanks for your responses.
> > > > > >> >
> > > > > >> > > I don't see a need of protobuf for this particular usecase
> > > > > >> >
> > > > > >> > If no one else feels strongly on this point, I am good with
> > using
> > > a
> > > > > >> POJO.
> > > > > >> >
> > > > > >> > > It doesn't matter if it's system-topic or not because it's
> > > > > >> > > configurable and the admin of the system can decide and
> > > configure
> > > > it
> > > > > >> > > according to the required persistent policy.
> > > > > >> >
> > > > > >> > On my first reading, it wasn't clear if there was only one
> topic
> > > > > >> > required for this feature. I now see that the topic is not
> tied
> > > to a
> > > > > >> > specific tenant or namespace. As such, we can avoid
> complicated
> > > > > >> > authorization questions by putting the required event topic(s)
> > > into
> > > > a
> > > > > >> > "system" tenant and namespace, by default. The `pulsar/system`
> > > > tenant
> > > > > >> > and namespace seem appropriate to me.
> > > > > >> >
> > > > > >> > > I would keep the system topic
> > > > > >> > > separate because this topic serves a specific purpose with
> > > > specific
> > > > > >> > schema,
> > > > > >> > > replication policy and retention policy.
> > > > > >> >
> > > > > >> > I think we need a more formal definition for system topics.
> This
> > > > topic
> > > > > >> > is exactly the kind of topic I would call a system topic: its
> > > > intended
> > > > > >> > producers and consumers are Pulsar components. However,
> because
> > > > > >> > this feature can live on a topic in a system namespace, we can
> > > avoid
> > > > > >> > the classification discussion for this PIP.
> > > > > >> >
> > > > > >> > > Source region will have a broker which will create a
> failover
> > > > > >> consumer on
> > > > > >> > > that topic and a broker with an active consumer will watch
> the
> > > > > >> metadata
> > > > > >> > > changes and publish the changes to the event topic.
> > > > > >> >
> > > > > >> > How do we designate the host broker? Is it manual? How does it
> > > work
> > > > > >> > when the host broker is removed from the cluster?
> > > > > >> >
> > > > > >> > If we collocate the active consumer with the broker hosting
> the
> > > > event
> > > > > >> > topic, can we skip creating the failover consumer?
> > > > > >> >
> > > > > >> > > PIP briefly talks about it but I will update the PIP with
> more
> > > > > >> > > explanation.
> > > > > >> >
> > > > > >> > I look forward to seeing more about this design for conflict
> > > > > resolution.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Michael
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> > > > dhabalia.me@gmail.com>
> > > > > >> > wrote:
> > > > > >> > >
> > > > > >> > > Please find my response inline.
> > > > > >> > >
> > > > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > > > > >> mmarshall@apache.org>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > I think this is a very appropriate direction to take
> > Pulsar's
> > > > > >> > > > geo-replication. Your proposal is essentially to make the
> > > > > >> > > > inter-cluster configuration event driven. This increases
> > fault
> > > > > >> > > > tolerance and better decouples clusters.
> > > > > >> > > >
> > > > > >> > > > Thank you for your detailed proposal. After reading
> through
> > > it,
> > > > I
> > > > > >> have
> > > > > >> > > > some questions :)
> > > > > >> > > >
> > > > > >> > > > 1. What do you think about using protobuf to define the
> > event
> > > > > >> > > > protocol? I know we already have a topic policy event
> stream
> > > > > >> > > > defined with Java POJOs, but since this feature is
> > > specifically
> > > > > >> > > > designed for egressing cloud providers, ensuring compact
> > data
> > > > > >> transfer
> > > > > >> > > > would keep egress costs down. Additionally, protobuf can
> > help
> > > > make
> > > > > >> it
> > > > > >> > > > clear that the schema is strict, should evolve
> thoughtfully,
> > > and
> > > > > >> > > > should be designed to work between clusters of different
> > > > versions.
> > > > > >> > > >
> > > > > >> > >
> > > > > >> > >  >>> I don't see a need of protobuf for this particular
> > usecase
> > > > > >> because
> > > > > >> > of
> > > > > >> > > two reasons:
> > > > > >> > >   >> a. policy changes don't generate huge traffic which
> could
> > > be
> > > > 1
> > > > > >> rps
> > > > > >> > b.
> > > > > >> > > and it doesn't need performance optimization.
> > > > > >> > >   >> It should be similar as storing policy in text instead
> > > > protobuf
> > > > > >> > which
> > > > > >> > > doesn't impact footprint size or performance due to limited
> > > number
> > > > > of
> > > > > >> > >  >> update operations and relatively less complexity. I
> agree
> > > that
> > > > > >> > protobuf
> > > > > >> > > could be another option but in this case it's not needed.
> > Also,
> > > > POJO
> > > > > >> > >  >> can also support schema and versioning.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > 2. In your view, which tenant/namespace will host
> > > > > >> > > > `metadataSyncEventTopic`? Will there be several of these
> > > topics
> > > > or
> > > > > >> is
> > > > > >> > > > it just hosted in a system tenant/namespace? This question
> > > gets
> > > > > back
> > > > > >> > > > to my questions about system topics on this mailing list
> > last
> > > > week
> > > > > >> > [0]. I
> > > > > >> > > > view this topic as a system topic, so we'd need to make
> sure
> > > > that
> > > > > it
> > > > > >> > > > has the right authorization rules and that it won't be
> > > affected
> > > > by
> > > > > >> > calls
> > > > > >> > > > like "clearNamespaceBacklog".
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >   >> It doesn't matter if it's system-topic or not because
> > it's
> > > > > >> > > configurable and the admin of the system can decide and
> > > configure
> > > > it
> > > > > >> > > according to the required persistent policy. I would keep
> the
> > > > system
> > > > > >> > topic
> > > > > >> > > separate because this topic serves a specific purpose with
> > > > specific
> > > > > >> > schema,
> > > > > >> > > replication policy and retention policy.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > 3. Which broker will host the metadata update publisher? I
> > > > assume
> > > > > we
> > > > > >> > > > want the producer to be collocated with the bundle that
> > hosts
> > > > the
> > > > > >> > > > event topic. How will this be coordinated?
> > > > > >> > > >
> > > > > >> > > >> It's already explained into PIP in section: "Event
> > publisher
> > > > and
> > > > > >> > handler"
> > > > > >> > > >> Every isolated cluster deployed on a separate cloud
> > platform
> > > > will
> > > > > >> > have a
> > > > > >> > > source region and part of replicated clusters for the event
> > > topic.
> > > > > The
> > > > > >> > > Source region will have a broker which will create a
> failover
> > > > > >> consumer on
> > > > > >> > > that topic and a broker with an active consumer will watch
> the
> > > > > >> metadata
> > > > > >> > > changes and publish the changes to the event topic.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the
> > > topic
> > > > > >> level
> > > > > >> > > > policies already have this feature? If so, is there a way
> to
> > > > > >> integrate
> > > > > >> > > > this feature with the existing topic policy feature?
> > > > > >> > > >
> > > > > >> > > >> Yes, ResourceType can be extensible to a topic as well.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > > 5. By decentralizing the metadata store, it looks like
> there
> > > is
> > > > a
> > > > > >> > > > chance for conflicts due to concurrent updates. How do we
> > > handle
> > > > > >> those
> > > > > >> > > > conflicts?
> > > > > >> > > >
> > > > > >> > > >>  PIP briefly talks about it but I will update the PIP
> with
> > > more
> > > > > >> > > explanation. MetadataChangeEvent contains source-cluster and
> > > > updated
> > > > > >> > time.
> > > > > >> > > Also, resources Tenant/Namespace will also contain
> > > lastUpdatedTime
> > > > > >> which
> > > > > >> > > will help to destination clusters to handle stale/duplicate
> > > events
> > > > > and
> > > > > >> > race
> > > > > >> > > conditions. Also, snapshot-sync an additional task helps all
> > > > > clusters
> > > > > >> to
> > > > > >> > be
> > > > > >> > > synced with each other eventually.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > > I'll also note that I previously proposed a system event
> > topic
> > > > > here
> > > > > >> > > > [1] and it was proposed again here [2]. Those features
> were
> > > for
> > > > > >> > > > different use cases, but ultimately looked very similar.
> In
> > my
> > > > > >> view, a
> > > > > >> > > > stream of system events is a very natural feature to
> expect
> > > in a
> > > > > >> > > > streaming technology. I wonder if there is a way to
> > generalize
> > > > > this
> > > > > >> > > > feature to fulfill local cluster consumers and
> > geo-replication
> > > > > >> > > > consumers. Even if this PIP only implements the
> > > geo-replication
> > > > > >> > > > portion of the feature, it'd be good to design it in an
> > > > extensible
> > > > > >> > fashion.
> > > > > >> > > >
> > > > > >> > >  >> I think answer (2) addresses this concern as well.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > > Thanks,
> > > > > >> > > > Michael
> > > > > >> > > >
> > > > > >> > > > [0]
> > > > > >>
> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > > >> > > > [1]
> > > > > >>
> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > > > >> > > > [2]
> > > > > >>
> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > > > > >> rdhabalia@apache.org>
> > > > > >> > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Hi,
> > > > > >> > > > >
> > > > > >> > > > > I would like to start a discussion about PIP-136: Sync
> > > Pulsar
> > > > > >> > policies
> > > > > >> > > > > across multiple clouds.
> > > > > >> > > > >
> > > > > >> > > > > PIP documentation:
> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > >> > > > >
> > > > > >> > > > > *Motivation*
> > > > > >> > > > > Apache Pulsar is a cloud-native, distributed messaging
> > > > framework
> > > > > >> > which
> > > > > >> > > > > natively provides geo-replication. Many organizations
> > deploy
> > > > > >> pulsar
> > > > > >> > > > > instances on-prem and on multiple different cloud
> > providers
> > > > and
> > > > > at
> > > > > >> > the
> > > > > >> > > > same
> > > > > >> > > > > time they would like to enable replication between
> > multiple
> > > > > >> clusters
> > > > > >> > > > > deployed in different cloud providers. Pulsar already
> > > provides
> > > > > >> > various
> > > > > >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions
> on
> > > > SNI)
> > > > > to
> > > > > >> > > > fulfill
> > > > > >> > > > > security requirements when brokers are deployed on
> > different
> > > > > >> security
> > > > > >> > > > zones
> > > > > >> > > > > connected with each other. However, sometimes it's not
> > > > possible
> > > > > to
> > > > > >> > share
> > > > > >> > > > > metadata-store (global zookeeper) between pulsar
> clusters
> > > > > >> deployed on
> > > > > >> > > > > separate cloud provider platforms, and synchronizing
> > > > > configuration
> > > > > >> > > > metadata
> > > > > >> > > > > (policies) can be a critical path to share
> > > > > tenant/namespace/topic
> > > > > >> > > > policies
> > > > > >> > > > > between clusters and administrate pulsar policies
> > uniformly
> > > > > across
> > > > > >> > all
> > > > > >> > > > > clusters. Therefore, we need a mechanism to sync
> > > configuration
> > > > > >> > metadata
> > > > > >> > > > > between clusters deployed on the different cloud
> > platforms.
> > > > > >> > > > >
> > > > > >> > > > > *Sync Pulsar policies across multiple clouds*
> > > > > >> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > >> > > > > Prototype git-hub-link
> > > > > >> > > > > <
> > > > > >> > > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > > > >> > > > >
> > > > > >> > > > > Thanks,
> > > > > >> > > > > Rajan
> > > > > >> > > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[PengHui Li <pe...@apache.org>]

Thanks for the explanation.

> yes, local policies doesn't need to be replicate to other clusters and it
will only replicate global policies which is shared across multiple
clusters such tenant/namespace's identity-creation, ACLs, replication
clusters, etc.

As described in this blog [1], section "Aggregation Replication".
Do we need to provide the ability for users to decide to replicate
the ACLs and replication cluster or not?

Currently, if users want to achieve "Aggregation Replication", it needs
multiple configuration stores. So they need to maintain the namespace,
partitioned topics in each cluster. A new namespace created in one cluster,
it need to create the namespace in other clusters if they want to replicate
data to those clusters.

After this proposal, they don't need to create a namespace for other
clusters,
Pulsar will help to replicate the configuration store changes to the
replicated cluster,
if the new created namespace with replication cluster A, B, and C in
cluster A, the
namespace will be replicated to B and C.

But for the ACLs and replication clusters, it should be controlled by users?
e.g. only replicate the namespace to B and C, but not the replication
clusters and ACLs.
So that we can achieve "Aggregation Replication" with this proposal.

> Topic that will be used to share policies across clusters is configurable
and it can be named anything. However, we should keep it a separate topic
as it requires unique schema and special handling to synchronize policies
across the clusters.

Yes, looks like currently we already have a mechanism to replicate policies.
We have a system topic under the namespace "__change_events", which only
has
topic policy changes for now. It can replicate anything under a namespace.
We have defined "EventType"[2] in PulsarEvent(structure used in
"__change_events").
And we already have a implementation for selective PulsarEvent
replication[3], and schema
replication[4].

So it looks like we can use the "__change_events" to replicate namespace
policies, and use a
new topic which belongs to a system namespace to replicate
tenant/namespace's identity-creation,
partitioned topic creation?

BTW, we already supported topic level replication cluster configuration[5],
looks like in this case,
the partitioned topic is created first in one cluster without replication
clusters first, after the replication
clusters changed, pulsar will replicate the partitioned topic to remote
cluster. The same mechanism is
required for non-partitioned topics(users might disabled the topic
auto-creation).

[1]
https://www.splunk.com/en_us/blog/devops/geo-replication-in-apache-pulsar-part-2-patterns-and-practices.html
[2]
https://github.com/apache/pulsar/blob/4dcb166e0bfcce7fc85fd8d59a25b881f6f9c6fa/pulsar-common/src/main/java/org/apache/pulsar/common/events/PulsarEvent.java#L36
[3]
https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
[4]
https://github.com/apache/pulsar/wiki/PIP-88%3A-Replicate-schemas-across-multiple
[5] https://github.com/apache/pulsar/pull/12136

Regards,
Penghui

On Tue, Mar 22, 2022 at 6:37 AM Rajan Dhabalia <rd...@apache.org> wrote:

> >> If it contains namespace policy replication, There are some policies no
> need to replicate to another cluster
> yes, local policies doesn't need to be replicate to other clusters and it
> will only replicate global policies which is shared across multiple
> clusters such tenant/namespace's identity-creation, ACLs, replication
> clusters, etc.
>
> >> The new partitioned topic also needs to be replicated to the remote
> cluster?
> Yes.
>
> Topic that will be used to share policies across clusters is configurable
> and it can be named anything. However, we should keep it a separate topic
> as it requires unique schema and special handling to synchronize policies
> across the clusters.
>
> Thanks,
> Rajan
>
> On Fri, Mar 18, 2022 at 9:12 PM PengHui Li <pe...@apache.org> wrote:
>
> > Hi Rajan,
> >
> > Thanks for the great proposal.
> >
> > Will all the namespace policies be replicated to the remote cluster?
> > I noticed the PIP title mentioned policies, but looks like from the
> > `MetadataChangeEvent`,
> > no namespaces policies defined. If it contains namespace policy
> > replication,
> > There are some policies no need to replicate to another cluster,
> > for example, the rate limiter, max producers/consumers limiter.
> > In
> >
> >
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> > ,
> > it introduced a --global option to provide ability to apply the policy in
> > global or local.
> >
> > The new partitioned topic also needs to be replicated to the remote
> > cluster?
> >
> > Currently, we already have a PulsarEvent struct to define the pulsar
> system
> > events,
> > Looks like we can use a unified event definition by PulsarEvent.
> >
> > Others look good to me.
> >
> > Regards,
> > Penghui
> >
> >
> >
> > On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <rd...@apache.org>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to start VOTE on PIP-136:
> > > > https://github.com/apache/pulsar/issues/13728
> > > >
> > > > Thanks,
> > > > Rajan
> > > >
> > > > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dhabalia.me@gmail.com
> >
> > > > wrote:
> > > >
> > > > >
> > > > > >> How do we designate the host broker? Is it manual? How does it
> > work
> > > > > when the host broker is removed from the cluster?
> > > > > No, it will not be manual but as I explained earlier a broker which
> > > has a
> > > > > failover consumer to consume remote events will be the publisher
> for
> > > > > metadata update. If that broker is removed then a new failover
> > > > > consumer/broker will be selected for the same.
> > > > >
> > > > > >> I look forward to seeing more about this design for conflict
> > > > resolution.
> > > > > Sure, I have updated PIP to handle such race condition:
> > > > https://github.com/apache/pulsar/issues/13728
> > > > >
> > > > >
> > > > > >> (1) scenarios where the Pulsar cluster operators and tenant
> admins
> > > > are
> > > > > different entities and tenants can be malicious, or more probably,
> > > write
> > > > > bad code that will produce malicious outcomes.
> > > > > I agree, Pulsar should have provision to prevent such scenarios
> where
> > > > > changes from one tenant in a cluster can impact other clusters.
> This
> > > PIP
> > > > > considers the tenant/admin will be the same at both the ends but
> that
> > > can
> > > > > not be true in all cases. We can add an enhancement later or we can
> > > > create
> > > > > a separate PIP to start discussion with the possible solutions.
> > > > >
> > > > > Thanks,
> > > > > Rajan
> > > > >
> > > > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com>
> wrote:
> > > > >
> > > > >> >On my first reading, it wasn't clear if there was only one topic
> > > > >> required for this feature. I now see that the topic is not tied
> to a
> > > > >> specific tenant or namespace. As such, we can avoid complicated
> > > > >> authorization questions by putting the required event topic(s)
> into
> > a
> > > > >> "system" tenant and namespace
> > > > >>
> > > > >> We should consider complicated questions. We can say why we chose
> > not
> > > to
> > > > >> address it, or why it does not apply. for a particular situation
> > > > >>
> > > > >> Many namespace policies are administered by tenants.  As such any
> > > tenant
> > > > >> can load this topic.  Is it possible for one abusive tenant to
> make
> > > your
> > > > >> system topic dysfunctional?
> > > > >>
> > > > >> Pulsar committers should think about
> > > > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> > > are
> > > > >> different entities and tenants can be malicious, or more probably,
> > > write
> > > > >> bad code that will produce malicious outcomes.
> > > > >> (2) whether the changes introduce  additional SPOFs into the
> > cluster.
> > > > >>
> > > > >> I don't think this PIP has those issues, but  as a matter of
> > > practice, I
> > > > >> would like to see backend/system PIPs consider these questions
> and
> > > > >> explicitly state the conclusions with rationale
> > > > >>
> > > > >> Joe
> > > > >>
> > > > >>
> > > > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <
> > mmarshall@apache.org
> > > >
> > > > >> wrote:
> > > > >>
> > > > >> > Thanks for your responses.
> > > > >> >
> > > > >> > > I don't see a need of protobuf for this particular usecase
> > > > >> >
> > > > >> > If no one else feels strongly on this point, I am good with
> using
> > a
> > > > >> POJO.
> > > > >> >
> > > > >> > > It doesn't matter if it's system-topic or not because it's
> > > > >> > > configurable and the admin of the system can decide and
> > configure
> > > it
> > > > >> > > according to the required persistent policy.
> > > > >> >
> > > > >> > On my first reading, it wasn't clear if there was only one topic
> > > > >> > required for this feature. I now see that the topic is not tied
> > to a
> > > > >> > specific tenant or namespace. As such, we can avoid complicated
> > > > >> > authorization questions by putting the required event topic(s)
> > into
> > > a
> > > > >> > "system" tenant and namespace, by default. The `pulsar/system`
> > > tenant
> > > > >> > and namespace seem appropriate to me.
> > > > >> >
> > > > >> > > I would keep the system topic
> > > > >> > > separate because this topic serves a specific purpose with
> > > specific
> > > > >> > schema,
> > > > >> > > replication policy and retention policy.
> > > > >> >
> > > > >> > I think we need a more formal definition for system topics. This
> > > topic
> > > > >> > is exactly the kind of topic I would call a system topic: its
> > > intended
> > > > >> > producers and consumers are Pulsar components. However, because
> > > > >> > this feature can live on a topic in a system namespace, we can
> > avoid
> > > > >> > the classification discussion for this PIP.
> > > > >> >
> > > > >> > > Source region will have a broker which will create a failover
> > > > >> consumer on
> > > > >> > > that topic and a broker with an active consumer will watch the
> > > > >> metadata
> > > > >> > > changes and publish the changes to the event topic.
> > > > >> >
> > > > >> > How do we designate the host broker? Is it manual? How does it
> > work
> > > > >> > when the host broker is removed from the cluster?
> > > > >> >
> > > > >> > If we collocate the active consumer with the broker hosting the
> > > event
> > > > >> > topic, can we skip creating the failover consumer?
> > > > >> >
> > > > >> > > PIP briefly talks about it but I will update the PIP with more
> > > > >> > > explanation.
> > > > >> >
> > > > >> > I look forward to seeing more about this design for conflict
> > > > resolution.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Michael
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> > > dhabalia.me@gmail.com>
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > Please find my response inline.
> > > > >> > >
> > > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > > > >> mmarshall@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > I think this is a very appropriate direction to take
> Pulsar's
> > > > >> > > > geo-replication. Your proposal is essentially to make the
> > > > >> > > > inter-cluster configuration event driven. This increases
> fault
> > > > >> > > > tolerance and better decouples clusters.
> > > > >> > > >
> > > > >> > > > Thank you for your detailed proposal. After reading through
> > it,
> > > I
> > > > >> have
> > > > >> > > > some questions :)
> > > > >> > > >
> > > > >> > > > 1. What do you think about using protobuf to define the
> event
> > > > >> > > > protocol? I know we already have a topic policy event stream
> > > > >> > > > defined with Java POJOs, but since this feature is
> > specifically
> > > > >> > > > designed for egressing cloud providers, ensuring compact
> data
> > > > >> transfer
> > > > >> > > > would keep egress costs down. Additionally, protobuf can
> help
> > > make
> > > > >> it
> > > > >> > > > clear that the schema is strict, should evolve thoughtfully,
> > and
> > > > >> > > > should be designed to work between clusters of different
> > > versions.
> > > > >> > > >
> > > > >> > >
> > > > >> > >  >>> I don't see a need of protobuf for this particular
> usecase
> > > > >> because
> > > > >> > of
> > > > >> > > two reasons:
> > > > >> > >   >> a. policy changes don't generate huge traffic which could
> > be
> > > 1
> > > > >> rps
> > > > >> > b.
> > > > >> > > and it doesn't need performance optimization.
> > > > >> > >   >> It should be similar as storing policy in text instead
> > > protobuf
> > > > >> > which
> > > > >> > > doesn't impact footprint size or performance due to limited
> > number
> > > > of
> > > > >> > >  >> update operations and relatively less complexity. I agree
> > that
> > > > >> > protobuf
> > > > >> > > could be another option but in this case it's not needed.
> Also,
> > > POJO
> > > > >> > >  >> can also support schema and versioning.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > 2. In your view, which tenant/namespace will host
> > > > >> > > > `metadataSyncEventTopic`? Will there be several of these
> > topics
> > > or
> > > > >> is
> > > > >> > > > it just hosted in a system tenant/namespace? This question
> > gets
> > > > back
> > > > >> > > > to my questions about system topics on this mailing list
> last
> > > week
> > > > >> > [0]. I
> > > > >> > > > view this topic as a system topic, so we'd need to make sure
> > > that
> > > > it
> > > > >> > > > has the right authorization rules and that it won't be
> > affected
> > > by
> > > > >> > calls
> > > > >> > > > like "clearNamespaceBacklog".
> > > > >> > >
> > > > >> > >
> > > > >> > >   >> It doesn't matter if it's system-topic or not because
> it's
> > > > >> > > configurable and the admin of the system can decide and
> > configure
> > > it
> > > > >> > > according to the required persistent policy. I would keep the
> > > system
> > > > >> > topic
> > > > >> > > separate because this topic serves a specific purpose with
> > > specific
> > > > >> > schema,
> > > > >> > > replication policy and retention policy.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > 3. Which broker will host the metadata update publisher? I
> > > assume
> > > > we
> > > > >> > > > want the producer to be collocated with the bundle that
> hosts
> > > the
> > > > >> > > > event topic. How will this be coordinated?
> > > > >> > > >
> > > > >> > > >> It's already explained into PIP in section: "Event
> publisher
> > > and
> > > > >> > handler"
> > > > >> > > >> Every isolated cluster deployed on a separate cloud
> platform
> > > will
> > > > >> > have a
> > > > >> > > source region and part of replicated clusters for the event
> > topic.
> > > > The
> > > > >> > > Source region will have a broker which will create a failover
> > > > >> consumer on
> > > > >> > > that topic and a broker with an active consumer will watch the
> > > > >> metadata
> > > > >> > > changes and publish the changes to the event topic.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the
> > topic
> > > > >> level
> > > > >> > > > policies already have this feature? If so, is there a way to
> > > > >> integrate
> > > > >> > > > this feature with the existing topic policy feature?
> > > > >> > > >
> > > > >> > > >> Yes, ResourceType can be extensible to a topic as well.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > >
> > > > >> > > > 5. By decentralizing the metadata store, it looks like there
> > is
> > > a
> > > > >> > > > chance for conflicts due to concurrent updates. How do we
> > handle
> > > > >> those
> > > > >> > > > conflicts?
> > > > >> > > >
> > > > >> > > >>  PIP briefly talks about it but I will update the PIP with
> > more
> > > > >> > > explanation. MetadataChangeEvent contains source-cluster and
> > > updated
> > > > >> > time.
> > > > >> > > Also, resources Tenant/Namespace will also contain
> > lastUpdatedTime
> > > > >> which
> > > > >> > > will help to destination clusters to handle stale/duplicate
> > events
> > > > and
> > > > >> > race
> > > > >> > > conditions. Also, snapshot-sync an additional task helps all
> > > > clusters
> > > > >> to
> > > > >> > be
> > > > >> > > synced with each other eventually.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > I'll also note that I previously proposed a system event
> topic
> > > > here
> > > > >> > > > [1] and it was proposed again here [2]. Those features were
> > for
> > > > >> > > > different use cases, but ultimately looked very similar. In
> my
> > > > >> view, a
> > > > >> > > > stream of system events is a very natural feature to expect
> > in a
> > > > >> > > > streaming technology. I wonder if there is a way to
> generalize
> > > > this
> > > > >> > > > feature to fulfill local cluster consumers and
> geo-replication
> > > > >> > > > consumers. Even if this PIP only implements the
> > geo-replication
> > > > >> > > > portion of the feature, it'd be good to design it in an
> > > extensible
> > > > >> > fashion.
> > > > >> > > >
> > > > >> > >  >> I think answer (2) addresses this concern as well.
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> > > > Thanks,
> > > > >> > > > Michael
> > > > >> > > >
> > > > >> > > > [0]
> > > > >> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > >> > > > [1]
> > > > >> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > > >> > > > [2]
> > > > >> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > > > >> rdhabalia@apache.org>
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > > Hi,
> > > > >> > > > >
> > > > >> > > > > I would like to start a discussion about PIP-136: Sync
> > Pulsar
> > > > >> > policies
> > > > >> > > > > across multiple clouds.
> > > > >> > > > >
> > > > >> > > > > PIP documentation:
> > > > https://github.com/apache/pulsar/issues/13728
> > > > >> > > > >
> > > > >> > > > > *Motivation*
> > > > >> > > > > Apache Pulsar is a cloud-native, distributed messaging
> > > framework
> > > > >> > which
> > > > >> > > > > natively provides geo-replication. Many organizations
> deploy
> > > > >> pulsar
> > > > >> > > > > instances on-prem and on multiple different cloud
> providers
> > > and
> > > > at
> > > > >> > the
> > > > >> > > > same
> > > > >> > > > > time they would like to enable replication between
> multiple
> > > > >> clusters
> > > > >> > > > > deployed in different cloud providers. Pulsar already
> > provides
> > > > >> > various
> > > > >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on
> > > SNI)
> > > > to
> > > > >> > > > fulfill
> > > > >> > > > > security requirements when brokers are deployed on
> different
> > > > >> security
> > > > >> > > > zones
> > > > >> > > > > connected with each other. However, sometimes it's not
> > > possible
> > > > to
> > > > >> > share
> > > > >> > > > > metadata-store (global zookeeper) between pulsar clusters
> > > > >> deployed on
> > > > >> > > > > separate cloud provider platforms, and synchronizing
> > > > configuration
> > > > >> > > > metadata
> > > > >> > > > > (policies) can be a critical path to share
> > > > tenant/namespace/topic
> > > > >> > > > policies
> > > > >> > > > > between clusters and administrate pulsar policies
> uniformly
> > > > across
> > > > >> > all
> > > > >> > > > > clusters. Therefore, we need a mechanism to sync
> > configuration
> > > > >> > metadata
> > > > >> > > > > between clusters deployed on the different cloud
> platforms.
> > > > >> > > > >
> > > > >> > > > > *Sync Pulsar policies across multiple clouds*
> > > > >> > > > > https://github.com/apache/pulsar/issues/13728
> > > > >> > > > > Prototype git-hub-link
> > > > >> > > > > <
> > > > >> > > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > > >> > > > >
> > > > >> > > > > Thanks,
> > > > >> > > > > Rajan
> > > > >> > > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <rd...@apache.org>]

>> If it contains namespace policy replication, There are some policies no
need to replicate to another cluster
yes, local policies doesn't need to be replicate to other clusters and it
will only replicate global policies which is shared across multiple
clusters such tenant/namespace's identity-creation, ACLs, replication
clusters, etc.

>> The new partitioned topic also needs to be replicated to the remote
cluster?
Yes.

Topic that will be used to share policies across clusters is configurable
and it can be named anything. However, we should keep it a separate topic
as it requires unique schema and special handling to synchronize policies
across the clusters.

Thanks,
Rajan

On Fri, Mar 18, 2022 at 9:12 PM PengHui Li <pe...@apache.org> wrote:

> Hi Rajan,
>
> Thanks for the great proposal.
>
> Will all the namespace policies be replicated to the remote cluster?
> I noticed the PIP title mentioned policies, but looks like from the
> `MetadataChangeEvent`,
> no namespaces policies defined. If it contains namespace policy
> replication,
> There are some policies no need to replicate to another cluster,
> for example, the rate limiter, max producers/consumers limiter.
> In
>
> https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
> ,
> it introduced a --global option to provide ability to apply the policy in
> global or local.
>
> The new partitioned topic also needs to be replicated to the remote
> cluster?
>
> Currently, we already have a PulsarEvent struct to define the pulsar system
> events,
> Looks like we can use a unified event definition by PulsarEvent.
>
> Others look good to me.
>
> Regards,
> Penghui
>
>
>
> On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com> wrote:
>
> > +1
> >
> > On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <rd...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I would like to start VOTE on PIP-136:
> > > https://github.com/apache/pulsar/issues/13728
> > >
> > > Thanks,
> > > Rajan
> > >
> > > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dh...@gmail.com>
> > > wrote:
> > >
> > > >
> > > > >> How do we designate the host broker? Is it manual? How does it
> work
> > > > when the host broker is removed from the cluster?
> > > > No, it will not be manual but as I explained earlier a broker which
> > has a
> > > > failover consumer to consume remote events will be the publisher for
> > > > metadata update. If that broker is removed then a new failover
> > > > consumer/broker will be selected for the same.
> > > >
> > > > >> I look forward to seeing more about this design for conflict
> > > resolution.
> > > > Sure, I have updated PIP to handle such race condition:
> > > https://github.com/apache/pulsar/issues/13728
> > > >
> > > >
> > > > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> > > are
> > > > different entities and tenants can be malicious, or more probably,
> > write
> > > > bad code that will produce malicious outcomes.
> > > > I agree, Pulsar should have provision to prevent such scenarios where
> > > > changes from one tenant in a cluster can impact other clusters. This
> > PIP
> > > > considers the tenant/admin will be the same at both the ends but that
> > can
> > > > not be true in all cases. We can add an enhancement later or we can
> > > create
> > > > a separate PIP to start discussion with the possible solutions.
> > > >
> > > > Thanks,
> > > > Rajan
> > > >
> > > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com> wrote:
> > > >
> > > >> >On my first reading, it wasn't clear if there was only one topic
> > > >> required for this feature. I now see that the topic is not tied to a
> > > >> specific tenant or namespace. As such, we can avoid complicated
> > > >> authorization questions by putting the required event topic(s) into
> a
> > > >> "system" tenant and namespace
> > > >>
> > > >> We should consider complicated questions. We can say why we chose
> not
> > to
> > > >> address it, or why it does not apply. for a particular situation
> > > >>
> > > >> Many namespace policies are administered by tenants.  As such any
> > tenant
> > > >> can load this topic.  Is it possible for one abusive tenant to make
> > your
> > > >> system topic dysfunctional?
> > > >>
> > > >> Pulsar committers should think about
> > > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> > are
> > > >> different entities and tenants can be malicious, or more probably,
> > write
> > > >> bad code that will produce malicious outcomes.
> > > >> (2) whether the changes introduce  additional SPOFs into the
> cluster.
> > > >>
> > > >> I don't think this PIP has those issues, but  as a matter of
> > practice, I
> > > >> would like to see backend/system PIPs consider these questions  and
> > > >> explicitly state the conclusions with rationale
> > > >>
> > > >> Joe
> > > >>
> > > >>
> > > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <
> mmarshall@apache.org
> > >
> > > >> wrote:
> > > >>
> > > >> > Thanks for your responses.
> > > >> >
> > > >> > > I don't see a need of protobuf for this particular usecase
> > > >> >
> > > >> > If no one else feels strongly on this point, I am good with using
> a
> > > >> POJO.
> > > >> >
> > > >> > > It doesn't matter if it's system-topic or not because it's
> > > >> > > configurable and the admin of the system can decide and
> configure
> > it
> > > >> > > according to the required persistent policy.
> > > >> >
> > > >> > On my first reading, it wasn't clear if there was only one topic
> > > >> > required for this feature. I now see that the topic is not tied
> to a
> > > >> > specific tenant or namespace. As such, we can avoid complicated
> > > >> > authorization questions by putting the required event topic(s)
> into
> > a
> > > >> > "system" tenant and namespace, by default. The `pulsar/system`
> > tenant
> > > >> > and namespace seem appropriate to me.
> > > >> >
> > > >> > > I would keep the system topic
> > > >> > > separate because this topic serves a specific purpose with
> > specific
> > > >> > schema,
> > > >> > > replication policy and retention policy.
> > > >> >
> > > >> > I think we need a more formal definition for system topics. This
> > topic
> > > >> > is exactly the kind of topic I would call a system topic: its
> > intended
> > > >> > producers and consumers are Pulsar components. However, because
> > > >> > this feature can live on a topic in a system namespace, we can
> avoid
> > > >> > the classification discussion for this PIP.
> > > >> >
> > > >> > > Source region will have a broker which will create a failover
> > > >> consumer on
> > > >> > > that topic and a broker with an active consumer will watch the
> > > >> metadata
> > > >> > > changes and publish the changes to the event topic.
> > > >> >
> > > >> > How do we designate the host broker? Is it manual? How does it
> work
> > > >> > when the host broker is removed from the cluster?
> > > >> >
> > > >> > If we collocate the active consumer with the broker hosting the
> > event
> > > >> > topic, can we skip creating the failover consumer?
> > > >> >
> > > >> > > PIP briefly talks about it but I will update the PIP with more
> > > >> > > explanation.
> > > >> >
> > > >> > I look forward to seeing more about this design for conflict
> > > resolution.
> > > >> >
> > > >> > Thanks,
> > > >> > Michael
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> > dhabalia.me@gmail.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > Please find my response inline.
> > > >> > >
> > > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > > >> mmarshall@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > I think this is a very appropriate direction to take Pulsar's
> > > >> > > > geo-replication. Your proposal is essentially to make the
> > > >> > > > inter-cluster configuration event driven. This increases fault
> > > >> > > > tolerance and better decouples clusters.
> > > >> > > >
> > > >> > > > Thank you for your detailed proposal. After reading through
> it,
> > I
> > > >> have
> > > >> > > > some questions :)
> > > >> > > >
> > > >> > > > 1. What do you think about using protobuf to define the event
> > > >> > > > protocol? I know we already have a topic policy event stream
> > > >> > > > defined with Java POJOs, but since this feature is
> specifically
> > > >> > > > designed for egressing cloud providers, ensuring compact data
> > > >> transfer
> > > >> > > > would keep egress costs down. Additionally, protobuf can help
> > make
> > > >> it
> > > >> > > > clear that the schema is strict, should evolve thoughtfully,
> and
> > > >> > > > should be designed to work between clusters of different
> > versions.
> > > >> > > >
> > > >> > >
> > > >> > >  >>> I don't see a need of protobuf for this particular usecase
> > > >> because
> > > >> > of
> > > >> > > two reasons:
> > > >> > >   >> a. policy changes don't generate huge traffic which could
> be
> > 1
> > > >> rps
> > > >> > b.
> > > >> > > and it doesn't need performance optimization.
> > > >> > >   >> It should be similar as storing policy in text instead
> > protobuf
> > > >> > which
> > > >> > > doesn't impact footprint size or performance due to limited
> number
> > > of
> > > >> > >  >> update operations and relatively less complexity. I agree
> that
> > > >> > protobuf
> > > >> > > could be another option but in this case it's not needed. Also,
> > POJO
> > > >> > >  >> can also support schema and versioning.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > >
> > > >> > > > 2. In your view, which tenant/namespace will host
> > > >> > > > `metadataSyncEventTopic`? Will there be several of these
> topics
> > or
> > > >> is
> > > >> > > > it just hosted in a system tenant/namespace? This question
> gets
> > > back
> > > >> > > > to my questions about system topics on this mailing list last
> > week
> > > >> > [0]. I
> > > >> > > > view this topic as a system topic, so we'd need to make sure
> > that
> > > it
> > > >> > > > has the right authorization rules and that it won't be
> affected
> > by
> > > >> > calls
> > > >> > > > like "clearNamespaceBacklog".
> > > >> > >
> > > >> > >
> > > >> > >   >> It doesn't matter if it's system-topic or not because it's
> > > >> > > configurable and the admin of the system can decide and
> configure
> > it
> > > >> > > according to the required persistent policy. I would keep the
> > system
> > > >> > topic
> > > >> > > separate because this topic serves a specific purpose with
> > specific
> > > >> > schema,
> > > >> > > replication policy and retention policy.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > >
> > > >> > > > 3. Which broker will host the metadata update publisher? I
> > assume
> > > we
> > > >> > > > want the producer to be collocated with the bundle that hosts
> > the
> > > >> > > > event topic. How will this be coordinated?
> > > >> > > >
> > > >> > > >> It's already explained into PIP in section: "Event publisher
> > and
> > > >> > handler"
> > > >> > > >> Every isolated cluster deployed on a separate cloud platform
> > will
> > > >> > have a
> > > >> > > source region and part of replicated clusters for the event
> topic.
> > > The
> > > >> > > Source region will have a broker which will create a failover
> > > >> consumer on
> > > >> > > that topic and a broker with an active consumer will watch the
> > > >> metadata
> > > >> > > changes and publish the changes to the event topic.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > >
> > > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the
> topic
> > > >> level
> > > >> > > > policies already have this feature? If so, is there a way to
> > > >> integrate
> > > >> > > > this feature with the existing topic policy feature?
> > > >> > > >
> > > >> > > >> Yes, ResourceType can be extensible to a topic as well.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > >
> > > >> > > > 5. By decentralizing the metadata store, it looks like there
> is
> > a
> > > >> > > > chance for conflicts due to concurrent updates. How do we
> handle
> > > >> those
> > > >> > > > conflicts?
> > > >> > > >
> > > >> > > >>  PIP briefly talks about it but I will update the PIP with
> more
> > > >> > > explanation. MetadataChangeEvent contains source-cluster and
> > updated
> > > >> > time.
> > > >> > > Also, resources Tenant/Namespace will also contain
> lastUpdatedTime
> > > >> which
> > > >> > > will help to destination clusters to handle stale/duplicate
> events
> > > and
> > > >> > race
> > > >> > > conditions. Also, snapshot-sync an additional task helps all
> > > clusters
> > > >> to
> > > >> > be
> > > >> > > synced with each other eventually.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > I'll also note that I previously proposed a system event topic
> > > here
> > > >> > > > [1] and it was proposed again here [2]. Those features were
> for
> > > >> > > > different use cases, but ultimately looked very similar. In my
> > > >> view, a
> > > >> > > > stream of system events is a very natural feature to expect
> in a
> > > >> > > > streaming technology. I wonder if there is a way to generalize
> > > this
> > > >> > > > feature to fulfill local cluster consumers and geo-replication
> > > >> > > > consumers. Even if this PIP only implements the
> geo-replication
> > > >> > > > portion of the feature, it'd be good to design it in an
> > extensible
> > > >> > fashion.
> > > >> > > >
> > > >> > >  >> I think answer (2) addresses this concern as well.
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > > Thanks,
> > > >> > > > Michael
> > > >> > > >
> > > >> > > > [0]
> > > >> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > >> > > > [1]
> > > >> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > >> > > > [2]
> > > >> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > >> > > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > > >> rdhabalia@apache.org>
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > Hi,
> > > >> > > > >
> > > >> > > > > I would like to start a discussion about PIP-136: Sync
> Pulsar
> > > >> > policies
> > > >> > > > > across multiple clouds.
> > > >> > > > >
> > > >> > > > > PIP documentation:
> > > https://github.com/apache/pulsar/issues/13728
> > > >> > > > >
> > > >> > > > > *Motivation*
> > > >> > > > > Apache Pulsar is a cloud-native, distributed messaging
> > framework
> > > >> > which
> > > >> > > > > natively provides geo-replication. Many organizations deploy
> > > >> pulsar
> > > >> > > > > instances on-prem and on multiple different cloud providers
> > and
> > > at
> > > >> > the
> > > >> > > > same
> > > >> > > > > time they would like to enable replication between multiple
> > > >> clusters
> > > >> > > > > deployed in different cloud providers. Pulsar already
> provides
> > > >> > various
> > > >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on
> > SNI)
> > > to
> > > >> > > > fulfill
> > > >> > > > > security requirements when brokers are deployed on different
> > > >> security
> > > >> > > > zones
> > > >> > > > > connected with each other. However, sometimes it's not
> > possible
> > > to
> > > >> > share
> > > >> > > > > metadata-store (global zookeeper) between pulsar clusters
> > > >> deployed on
> > > >> > > > > separate cloud provider platforms, and synchronizing
> > > configuration
> > > >> > > > metadata
> > > >> > > > > (policies) can be a critical path to share
> > > tenant/namespace/topic
> > > >> > > > policies
> > > >> > > > > between clusters and administrate pulsar policies uniformly
> > > across
> > > >> > all
> > > >> > > > > clusters. Therefore, we need a mechanism to sync
> configuration
> > > >> > metadata
> > > >> > > > > between clusters deployed on the different cloud platforms.
> > > >> > > > >
> > > >> > > > > *Sync Pulsar policies across multiple clouds*
> > > >> > > > > https://github.com/apache/pulsar/issues/13728
> > > >> > > > > Prototype git-hub-link
> > > >> > > > > <
> > > >> > > >
> > > >> >
> > > >>
> > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Rajan
> > > >> > > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[PengHui Li <pe...@apache.org>]

Hi Rajan,

Thanks for the great proposal.

Will all the namespace policies be replicated to the remote cluster?
I noticed the PIP title mentioned policies, but looks like from the
`MetadataChangeEvent`,
no namespaces policies defined. If it contains namespace policy replication,
There are some policies no need to replicate to another cluster,
for example, the rate limiter, max producers/consumers limiter.
In
https://github.com/apache/pulsar/wiki/PIP-92%3A-Topic-policy-across-multiple-clusters
,
it introduced a --global option to provide ability to apply the policy in
global or local.

The new partitioned topic also needs to be replicated to the remote cluster?

Currently, we already have a PulsarEvent struct to define the pulsar system
events,
Looks like we can use a unified event definition by PulsarEvent.

Others look good to me.

Regards,
Penghui



On Sat, Mar 19, 2022 at 1:32 AM Joe F <jo...@gmail.com> wrote:

> +1
>
> On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <rd...@apache.org>
> wrote:
>
> > Hi,
> >
> > I would like to start VOTE on PIP-136:
> > https://github.com/apache/pulsar/issues/13728
> >
> > Thanks,
> > Rajan
> >
> > On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dh...@gmail.com>
> > wrote:
> >
> > >
> > > >> How do we designate the host broker? Is it manual? How does it work
> > > when the host broker is removed from the cluster?
> > > No, it will not be manual but as I explained earlier a broker which
> has a
> > > failover consumer to consume remote events will be the publisher for
> > > metadata update. If that broker is removed then a new failover
> > > consumer/broker will be selected for the same.
> > >
> > > >> I look forward to seeing more about this design for conflict
> > resolution.
> > > Sure, I have updated PIP to handle such race condition:
> > https://github.com/apache/pulsar/issues/13728
> > >
> > >
> > > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> > are
> > > different entities and tenants can be malicious, or more probably,
> write
> > > bad code that will produce malicious outcomes.
> > > I agree, Pulsar should have provision to prevent such scenarios where
> > > changes from one tenant in a cluster can impact other clusters. This
> PIP
> > > considers the tenant/admin will be the same at both the ends but that
> can
> > > not be true in all cases. We can add an enhancement later or we can
> > create
> > > a separate PIP to start discussion with the possible solutions.
> > >
> > > Thanks,
> > > Rajan
> > >
> > > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com> wrote:
> > >
> > >> >On my first reading, it wasn't clear if there was only one topic
> > >> required for this feature. I now see that the topic is not tied to a
> > >> specific tenant or namespace. As such, we can avoid complicated
> > >> authorization questions by putting the required event topic(s) into a
> > >> "system" tenant and namespace
> > >>
> > >> We should consider complicated questions. We can say why we chose not
> to
> > >> address it, or why it does not apply. for a particular situation
> > >>
> > >> Many namespace policies are administered by tenants.  As such any
> tenant
> > >> can load this topic.  Is it possible for one abusive tenant to make
> your
> > >> system topic dysfunctional?
> > >>
> > >> Pulsar committers should think about
> > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> are
> > >> different entities and tenants can be malicious, or more probably,
> write
> > >> bad code that will produce malicious outcomes.
> > >> (2) whether the changes introduce  additional SPOFs into the cluster.
> > >>
> > >> I don't think this PIP has those issues, but  as a matter of
> practice, I
> > >> would like to see backend/system PIPs consider these questions  and
> > >> explicitly state the conclusions with rationale
> > >>
> > >> Joe
> > >>
> > >>
> > >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mmarshall@apache.org
> >
> > >> wrote:
> > >>
> > >> > Thanks for your responses.
> > >> >
> > >> > > I don't see a need of protobuf for this particular usecase
> > >> >
> > >> > If no one else feels strongly on this point, I am good with using a
> > >> POJO.
> > >> >
> > >> > > It doesn't matter if it's system-topic or not because it's
> > >> > > configurable and the admin of the system can decide and configure
> it
> > >> > > according to the required persistent policy.
> > >> >
> > >> > On my first reading, it wasn't clear if there was only one topic
> > >> > required for this feature. I now see that the topic is not tied to a
> > >> > specific tenant or namespace. As such, we can avoid complicated
> > >> > authorization questions by putting the required event topic(s) into
> a
> > >> > "system" tenant and namespace, by default. The `pulsar/system`
> tenant
> > >> > and namespace seem appropriate to me.
> > >> >
> > >> > > I would keep the system topic
> > >> > > separate because this topic serves a specific purpose with
> specific
> > >> > schema,
> > >> > > replication policy and retention policy.
> > >> >
> > >> > I think we need a more formal definition for system topics. This
> topic
> > >> > is exactly the kind of topic I would call a system topic: its
> intended
> > >> > producers and consumers are Pulsar components. However, because
> > >> > this feature can live on a topic in a system namespace, we can avoid
> > >> > the classification discussion for this PIP.
> > >> >
> > >> > > Source region will have a broker which will create a failover
> > >> consumer on
> > >> > > that topic and a broker with an active consumer will watch the
> > >> metadata
> > >> > > changes and publish the changes to the event topic.
> > >> >
> > >> > How do we designate the host broker? Is it manual? How does it work
> > >> > when the host broker is removed from the cluster?
> > >> >
> > >> > If we collocate the active consumer with the broker hosting the
> event
> > >> > topic, can we skip creating the failover consumer?
> > >> >
> > >> > > PIP briefly talks about it but I will update the PIP with more
> > >> > > explanation.
> > >> >
> > >> > I look forward to seeing more about this design for conflict
> > resolution.
> > >> >
> > >> > Thanks,
> > >> > Michael
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <
> dhabalia.me@gmail.com>
> > >> > wrote:
> > >> > >
> > >> > > Please find my response inline.
> > >> > >
> > >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> > >> mmarshall@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > > I think this is a very appropriate direction to take Pulsar's
> > >> > > > geo-replication. Your proposal is essentially to make the
> > >> > > > inter-cluster configuration event driven. This increases fault
> > >> > > > tolerance and better decouples clusters.
> > >> > > >
> > >> > > > Thank you for your detailed proposal. After reading through it,
> I
> > >> have
> > >> > > > some questions :)
> > >> > > >
> > >> > > > 1. What do you think about using protobuf to define the event
> > >> > > > protocol? I know we already have a topic policy event stream
> > >> > > > defined with Java POJOs, but since this feature is specifically
> > >> > > > designed for egressing cloud providers, ensuring compact data
> > >> transfer
> > >> > > > would keep egress costs down. Additionally, protobuf can help
> make
> > >> it
> > >> > > > clear that the schema is strict, should evolve thoughtfully, and
> > >> > > > should be designed to work between clusters of different
> versions.
> > >> > > >
> > >> > >
> > >> > >  >>> I don't see a need of protobuf for this particular usecase
> > >> because
> > >> > of
> > >> > > two reasons:
> > >> > >   >> a. policy changes don't generate huge traffic which could be
> 1
> > >> rps
> > >> > b.
> > >> > > and it doesn't need performance optimization.
> > >> > >   >> It should be similar as storing policy in text instead
> protobuf
> > >> > which
> > >> > > doesn't impact footprint size or performance due to limited number
> > of
> > >> > >  >> update operations and relatively less complexity. I agree that
> > >> > protobuf
> > >> > > could be another option but in this case it's not needed. Also,
> POJO
> > >> > >  >> can also support schema and versioning.
> > >> > >
> > >> > >
> > >> > >
> > >> > > >
> > >> > > > 2. In your view, which tenant/namespace will host
> > >> > > > `metadataSyncEventTopic`? Will there be several of these topics
> or
> > >> is
> > >> > > > it just hosted in a system tenant/namespace? This question gets
> > back
> > >> > > > to my questions about system topics on this mailing list last
> week
> > >> > [0]. I
> > >> > > > view this topic as a system topic, so we'd need to make sure
> that
> > it
> > >> > > > has the right authorization rules and that it won't be affected
> by
> > >> > calls
> > >> > > > like "clearNamespaceBacklog".
> > >> > >
> > >> > >
> > >> > >   >> It doesn't matter if it's system-topic or not because it's
> > >> > > configurable and the admin of the system can decide and configure
> it
> > >> > > according to the required persistent policy. I would keep the
> system
> > >> > topic
> > >> > > separate because this topic serves a specific purpose with
> specific
> > >> > schema,
> > >> > > replication policy and retention policy.
> > >> > >
> > >> > >
> > >> > >
> > >> > > >
> > >> > > > 3. Which broker will host the metadata update publisher? I
> assume
> > we
> > >> > > > want the producer to be collocated with the bundle that hosts
> the
> > >> > > > event topic. How will this be coordinated?
> > >> > > >
> > >> > > >> It's already explained into PIP in section: "Event publisher
> and
> > >> > handler"
> > >> > > >> Every isolated cluster deployed on a separate cloud platform
> will
> > >> > have a
> > >> > > source region and part of replicated clusters for the event topic.
> > The
> > >> > > Source region will have a broker which will create a failover
> > >> consumer on
> > >> > > that topic and a broker with an active consumer will watch the
> > >> metadata
> > >> > > changes and publish the changes to the event topic.
> > >> > >
> > >> > >
> > >> > >
> > >> > > >
> > >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the topic
> > >> level
> > >> > > > policies already have this feature? If so, is there a way to
> > >> integrate
> > >> > > > this feature with the existing topic policy feature?
> > >> > > >
> > >> > > >> Yes, ResourceType can be extensible to a topic as well.
> > >> > >
> > >> > >
> > >> > >
> > >> > > >
> > >> > > > 5. By decentralizing the metadata store, it looks like there is
> a
> > >> > > > chance for conflicts due to concurrent updates. How do we handle
> > >> those
> > >> > > > conflicts?
> > >> > > >
> > >> > > >>  PIP briefly talks about it but I will update the PIP with more
> > >> > > explanation. MetadataChangeEvent contains source-cluster and
> updated
> > >> > time.
> > >> > > Also, resources Tenant/Namespace will also contain lastUpdatedTime
> > >> which
> > >> > > will help to destination clusters to handle stale/duplicate events
> > and
> > >> > race
> > >> > > conditions. Also, snapshot-sync an additional task helps all
> > clusters
> > >> to
> > >> > be
> > >> > > synced with each other eventually.
> > >> > >
> > >> > >
> > >> > >
> > >> > > > I'll also note that I previously proposed a system event topic
> > here
> > >> > > > [1] and it was proposed again here [2]. Those features were for
> > >> > > > different use cases, but ultimately looked very similar. In my
> > >> view, a
> > >> > > > stream of system events is a very natural feature to expect in a
> > >> > > > streaming technology. I wonder if there is a way to generalize
> > this
> > >> > > > feature to fulfill local cluster consumers and geo-replication
> > >> > > > consumers. Even if this PIP only implements the geo-replication
> > >> > > > portion of the feature, it'd be good to design it in an
> extensible
> > >> > fashion.
> > >> > > >
> > >> > >  >> I think answer (2) addresses this concern as well.
> > >> > >
> > >> > >
> > >> > >
> > >> > > > Thanks,
> > >> > > > Michael
> > >> > > >
> > >> > > > [0]
> > >> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > >> > > > [1]
> > >> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > >> > > > [2]
> > >> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> > >> rdhabalia@apache.org>
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I would like to start a discussion about PIP-136: Sync Pulsar
> > >> > policies
> > >> > > > > across multiple clouds.
> > >> > > > >
> > >> > > > > PIP documentation:
> > https://github.com/apache/pulsar/issues/13728
> > >> > > > >
> > >> > > > > *Motivation*
> > >> > > > > Apache Pulsar is a cloud-native, distributed messaging
> framework
> > >> > which
> > >> > > > > natively provides geo-replication. Many organizations deploy
> > >> pulsar
> > >> > > > > instances on-prem and on multiple different cloud providers
> and
> > at
> > >> > the
> > >> > > > same
> > >> > > > > time they would like to enable replication between multiple
> > >> clusters
> > >> > > > > deployed in different cloud providers. Pulsar already provides
> > >> > various
> > >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on
> SNI)
> > to
> > >> > > > fulfill
> > >> > > > > security requirements when brokers are deployed on different
> > >> security
> > >> > > > zones
> > >> > > > > connected with each other. However, sometimes it's not
> possible
> > to
> > >> > share
> > >> > > > > metadata-store (global zookeeper) between pulsar clusters
> > >> deployed on
> > >> > > > > separate cloud provider platforms, and synchronizing
> > configuration
> > >> > > > metadata
> > >> > > > > (policies) can be a critical path to share
> > tenant/namespace/topic
> > >> > > > policies
> > >> > > > > between clusters and administrate pulsar policies uniformly
> > across
> > >> > all
> > >> > > > > clusters. Therefore, we need a mechanism to sync configuration
> > >> > metadata
> > >> > > > > between clusters deployed on the different cloud platforms.
> > >> > > > >
> > >> > > > > *Sync Pulsar policies across multiple clouds*
> > >> > > > > https://github.com/apache/pulsar/issues/13728
> > >> > > > > Prototype git-hub-link
> > >> > > > > <
> > >> > > >
> > >> >
> > >>
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Rajan
> > >> > > >
> > >> >
> > >>
> > >
> >
>

Re: [VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[Joe F <jo...@gmail.com>]

+1

On Thu, Mar 17, 2022 at 12:07 PM Rajan Dhabalia <rd...@apache.org>
wrote:

> Hi,
>
> I would like to start VOTE on PIP-136:
> https://github.com/apache/pulsar/issues/13728
>
> Thanks,
> Rajan
>
> On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dh...@gmail.com>
> wrote:
>
> >
> > >> How do we designate the host broker? Is it manual? How does it work
> > when the host broker is removed from the cluster?
> > No, it will not be manual but as I explained earlier a broker which has a
> > failover consumer to consume remote events will be the publisher for
> > metadata update. If that broker is removed then a new failover
> > consumer/broker will be selected for the same.
> >
> > >> I look forward to seeing more about this design for conflict
> resolution.
> > Sure, I have updated PIP to handle such race condition:
> https://github.com/apache/pulsar/issues/13728
> >
> >
> > >> (1) scenarios where the Pulsar cluster operators and tenant admins
> are
> > different entities and tenants can be malicious, or more probably, write
> > bad code that will produce malicious outcomes.
> > I agree, Pulsar should have provision to prevent such scenarios where
> > changes from one tenant in a cluster can impact other clusters. This PIP
> > considers the tenant/admin will be the same at both the ends but that can
> > not be true in all cases. We can add an enhancement later or we can
> create
> > a separate PIP to start discussion with the possible solutions.
> >
> > Thanks,
> > Rajan
> >
> > On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com> wrote:
> >
> >> >On my first reading, it wasn't clear if there was only one topic
> >> required for this feature. I now see that the topic is not tied to a
> >> specific tenant or namespace. As such, we can avoid complicated
> >> authorization questions by putting the required event topic(s) into a
> >> "system" tenant and namespace
> >>
> >> We should consider complicated questions. We can say why we chose not to
> >> address it, or why it does not apply. for a particular situation
> >>
> >> Many namespace policies are administered by tenants.  As such any tenant
> >> can load this topic.  Is it possible for one abusive tenant to make your
> >> system topic dysfunctional?
> >>
> >> Pulsar committers should think about
> >> (1) scenarios where the Pulsar cluster operators and tenant admins  are
> >> different entities and tenants can be malicious, or more probably, write
> >> bad code that will produce malicious outcomes.
> >> (2) whether the changes introduce  additional SPOFs into the cluster.
> >>
> >> I don't think this PIP has those issues, but  as a matter of practice, I
> >> would like to see backend/system PIPs consider these questions  and
> >> explicitly state the conclusions with rationale
> >>
> >> Joe
> >>
> >>
> >> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mm...@apache.org>
> >> wrote:
> >>
> >> > Thanks for your responses.
> >> >
> >> > > I don't see a need of protobuf for this particular usecase
> >> >
> >> > If no one else feels strongly on this point, I am good with using a
> >> POJO.
> >> >
> >> > > It doesn't matter if it's system-topic or not because it's
> >> > > configurable and the admin of the system can decide and configure it
> >> > > according to the required persistent policy.
> >> >
> >> > On my first reading, it wasn't clear if there was only one topic
> >> > required for this feature. I now see that the topic is not tied to a
> >> > specific tenant or namespace. As such, we can avoid complicated
> >> > authorization questions by putting the required event topic(s) into a
> >> > "system" tenant and namespace, by default. The `pulsar/system` tenant
> >> > and namespace seem appropriate to me.
> >> >
> >> > > I would keep the system topic
> >> > > separate because this topic serves a specific purpose with specific
> >> > schema,
> >> > > replication policy and retention policy.
> >> >
> >> > I think we need a more formal definition for system topics. This topic
> >> > is exactly the kind of topic I would call a system topic: its intended
> >> > producers and consumers are Pulsar components. However, because
> >> > this feature can live on a topic in a system namespace, we can avoid
> >> > the classification discussion for this PIP.
> >> >
> >> > > Source region will have a broker which will create a failover
> >> consumer on
> >> > > that topic and a broker with an active consumer will watch the
> >> metadata
> >> > > changes and publish the changes to the event topic.
> >> >
> >> > How do we designate the host broker? Is it manual? How does it work
> >> > when the host broker is removed from the cluster?
> >> >
> >> > If we collocate the active consumer with the broker hosting the event
> >> > topic, can we skip creating the failover consumer?
> >> >
> >> > > PIP briefly talks about it but I will update the PIP with more
> >> > > explanation.
> >> >
> >> > I look forward to seeing more about this design for conflict
> resolution.
> >> >
> >> > Thanks,
> >> > Michael
> >> >
> >> >
> >> >
> >> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dh...@gmail.com>
> >> > wrote:
> >> > >
> >> > > Please find my response inline.
> >> > >
> >> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
> >> mmarshall@apache.org>
> >> > > wrote:
> >> > >
> >> > > > I think this is a very appropriate direction to take Pulsar's
> >> > > > geo-replication. Your proposal is essentially to make the
> >> > > > inter-cluster configuration event driven. This increases fault
> >> > > > tolerance and better decouples clusters.
> >> > > >
> >> > > > Thank you for your detailed proposal. After reading through it, I
> >> have
> >> > > > some questions :)
> >> > > >
> >> > > > 1. What do you think about using protobuf to define the event
> >> > > > protocol? I know we already have a topic policy event stream
> >> > > > defined with Java POJOs, but since this feature is specifically
> >> > > > designed for egressing cloud providers, ensuring compact data
> >> transfer
> >> > > > would keep egress costs down. Additionally, protobuf can help make
> >> it
> >> > > > clear that the schema is strict, should evolve thoughtfully, and
> >> > > > should be designed to work between clusters of different versions.
> >> > > >
> >> > >
> >> > >  >>> I don't see a need of protobuf for this particular usecase
> >> because
> >> > of
> >> > > two reasons:
> >> > >   >> a. policy changes don't generate huge traffic which could be 1
> >> rps
> >> > b.
> >> > > and it doesn't need performance optimization.
> >> > >   >> It should be similar as storing policy in text instead protobuf
> >> > which
> >> > > doesn't impact footprint size or performance due to limited number
> of
> >> > >  >> update operations and relatively less complexity. I agree that
> >> > protobuf
> >> > > could be another option but in this case it's not needed. Also, POJO
> >> > >  >> can also support schema and versioning.
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > > > 2. In your view, which tenant/namespace will host
> >> > > > `metadataSyncEventTopic`? Will there be several of these topics or
> >> is
> >> > > > it just hosted in a system tenant/namespace? This question gets
> back
> >> > > > to my questions about system topics on this mailing list last week
> >> > [0]. I
> >> > > > view this topic as a system topic, so we'd need to make sure that
> it
> >> > > > has the right authorization rules and that it won't be affected by
> >> > calls
> >> > > > like "clearNamespaceBacklog".
> >> > >
> >> > >
> >> > >   >> It doesn't matter if it's system-topic or not because it's
> >> > > configurable and the admin of the system can decide and configure it
> >> > > according to the required persistent policy. I would keep the system
> >> > topic
> >> > > separate because this topic serves a specific purpose with specific
> >> > schema,
> >> > > replication policy and retention policy.
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > > > 3. Which broker will host the metadata update publisher? I assume
> we
> >> > > > want the producer to be collocated with the bundle that hosts the
> >> > > > event topic. How will this be coordinated?
> >> > > >
> >> > > >> It's already explained into PIP in section: "Event publisher and
> >> > handler"
> >> > > >> Every isolated cluster deployed on a separate cloud platform will
> >> > have a
> >> > > source region and part of replicated clusters for the event topic.
> The
> >> > > Source region will have a broker which will create a failover
> >> consumer on
> >> > > that topic and a broker with an active consumer will watch the
> >> metadata
> >> > > changes and publish the changes to the event topic.
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > > > 4. Why isn't a topic a `ResourceType`? Is this because the topic
> >> level
> >> > > > policies already have this feature? If so, is there a way to
> >> integrate
> >> > > > this feature with the existing topic policy feature?
> >> > > >
> >> > > >> Yes, ResourceType can be extensible to a topic as well.
> >> > >
> >> > >
> >> > >
> >> > > >
> >> > > > 5. By decentralizing the metadata store, it looks like there is a
> >> > > > chance for conflicts due to concurrent updates. How do we handle
> >> those
> >> > > > conflicts?
> >> > > >
> >> > > >>  PIP briefly talks about it but I will update the PIP with more
> >> > > explanation. MetadataChangeEvent contains source-cluster and updated
> >> > time.
> >> > > Also, resources Tenant/Namespace will also contain lastUpdatedTime
> >> which
> >> > > will help to destination clusters to handle stale/duplicate events
> and
> >> > race
> >> > > conditions. Also, snapshot-sync an additional task helps all
> clusters
> >> to
> >> > be
> >> > > synced with each other eventually.
> >> > >
> >> > >
> >> > >
> >> > > > I'll also note that I previously proposed a system event topic
> here
> >> > > > [1] and it was proposed again here [2]. Those features were for
> >> > > > different use cases, but ultimately looked very similar. In my
> >> view, a
> >> > > > stream of system events is a very natural feature to expect in a
> >> > > > streaming technology. I wonder if there is a way to generalize
> this
> >> > > > feature to fulfill local cluster consumers and geo-replication
> >> > > > consumers. Even if this PIP only implements the geo-replication
> >> > > > portion of the feature, it'd be good to design it in an extensible
> >> > fashion.
> >> > > >
> >> > >  >> I think answer (2) addresses this concern as well.
> >> > >
> >> > >
> >> > >
> >> > > > Thanks,
> >> > > > Michael
> >> > > >
> >> > > > [0]
> >> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> >> > > > [1]
> >> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> >> > > > [2]
> >> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> >> rdhabalia@apache.org>
> >> > > > wrote:
> >> > > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > I would like to start a discussion about PIP-136: Sync Pulsar
> >> > policies
> >> > > > > across multiple clouds.
> >> > > > >
> >> > > > > PIP documentation:
> https://github.com/apache/pulsar/issues/13728
> >> > > > >
> >> > > > > *Motivation*
> >> > > > > Apache Pulsar is a cloud-native, distributed messaging framework
> >> > which
> >> > > > > natively provides geo-replication. Many organizations deploy
> >> pulsar
> >> > > > > instances on-prem and on multiple different cloud providers and
> at
> >> > the
> >> > > > same
> >> > > > > time they would like to enable replication between multiple
> >> clusters
> >> > > > > deployed in different cloud providers. Pulsar already provides
> >> > various
> >> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI)
> to
> >> > > > fulfill
> >> > > > > security requirements when brokers are deployed on different
> >> security
> >> > > > zones
> >> > > > > connected with each other. However, sometimes it's not possible
> to
> >> > share
> >> > > > > metadata-store (global zookeeper) between pulsar clusters
> >> deployed on
> >> > > > > separate cloud provider platforms, and synchronizing
> configuration
> >> > > > metadata
> >> > > > > (policies) can be a critical path to share
> tenant/namespace/topic
> >> > > > policies
> >> > > > > between clusters and administrate pulsar policies uniformly
> across
> >> > all
> >> > > > > clusters. Therefore, we need a mechanism to sync configuration
> >> > metadata
> >> > > > > between clusters deployed on the different cloud platforms.
> >> > > > >
> >> > > > > *Sync Pulsar policies across multiple clouds*
> >> > > > > https://github.com/apache/pulsar/issues/13728
> >> > > > > Prototype git-hub-link
> >> > > > > <
> >> > > >
> >> >
> >>
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Rajan
> >> > > >
> >> >
> >>
> >
>

[VOTE] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <rd...@apache.org>]

Hi,

I would like to start VOTE on PIP-136:
https://github.com/apache/pulsar/issues/13728

Thanks,
Rajan

On Tue, Feb 8, 2022 at 4:58 PM Rajan Dhabalia <dh...@gmail.com> wrote:

>
> >> How do we designate the host broker? Is it manual? How does it work
> when the host broker is removed from the cluster?
> No, it will not be manual but as I explained earlier a broker which has a
> failover consumer to consume remote events will be the publisher for
> metadata update. If that broker is removed then a new failover
> consumer/broker will be selected for the same.
>
> >> I look forward to seeing more about this design for conflict resolution.
> Sure, I have updated PIP to handle such race condition: https://github.com/apache/pulsar/issues/13728
>
>
> >> (1) scenarios where the Pulsar cluster operators and tenant admins  are
> different entities and tenants can be malicious, or more probably, write
> bad code that will produce malicious outcomes.
> I agree, Pulsar should have provision to prevent such scenarios where
> changes from one tenant in a cluster can impact other clusters. This PIP
> considers the tenant/admin will be the same at both the ends but that can
> not be true in all cases. We can add an enhancement later or we can create
> a separate PIP to start discussion with the possible solutions.
>
> Thanks,
> Rajan
>
> On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com> wrote:
>
>> >On my first reading, it wasn't clear if there was only one topic
>> required for this feature. I now see that the topic is not tied to a
>> specific tenant or namespace. As such, we can avoid complicated
>> authorization questions by putting the required event topic(s) into a
>> "system" tenant and namespace
>>
>> We should consider complicated questions. We can say why we chose not to
>> address it, or why it does not apply. for a particular situation
>>
>> Many namespace policies are administered by tenants.  As such any tenant
>> can load this topic.  Is it possible for one abusive tenant to make your
>> system topic dysfunctional?
>>
>> Pulsar committers should think about
>> (1) scenarios where the Pulsar cluster operators and tenant admins  are
>> different entities and tenants can be malicious, or more probably, write
>> bad code that will produce malicious outcomes.
>> (2) whether the changes introduce  additional SPOFs into the cluster.
>>
>> I don't think this PIP has those issues, but  as a matter of practice, I
>> would like to see backend/system PIPs consider these questions  and
>> explicitly state the conclusions with rationale
>>
>> Joe
>>
>>
>> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mm...@apache.org>
>> wrote:
>>
>> > Thanks for your responses.
>> >
>> > > I don't see a need of protobuf for this particular usecase
>> >
>> > If no one else feels strongly on this point, I am good with using a
>> POJO.
>> >
>> > > It doesn't matter if it's system-topic or not because it's
>> > > configurable and the admin of the system can decide and configure it
>> > > according to the required persistent policy.
>> >
>> > On my first reading, it wasn't clear if there was only one topic
>> > required for this feature. I now see that the topic is not tied to a
>> > specific tenant or namespace. As such, we can avoid complicated
>> > authorization questions by putting the required event topic(s) into a
>> > "system" tenant and namespace, by default. The `pulsar/system` tenant
>> > and namespace seem appropriate to me.
>> >
>> > > I would keep the system topic
>> > > separate because this topic serves a specific purpose with specific
>> > schema,
>> > > replication policy and retention policy.
>> >
>> > I think we need a more formal definition for system topics. This topic
>> > is exactly the kind of topic I would call a system topic: its intended
>> > producers and consumers are Pulsar components. However, because
>> > this feature can live on a topic in a system namespace, we can avoid
>> > the classification discussion for this PIP.
>> >
>> > > Source region will have a broker which will create a failover
>> consumer on
>> > > that topic and a broker with an active consumer will watch the
>> metadata
>> > > changes and publish the changes to the event topic.
>> >
>> > How do we designate the host broker? Is it manual? How does it work
>> > when the host broker is removed from the cluster?
>> >
>> > If we collocate the active consumer with the broker hosting the event
>> > topic, can we skip creating the failover consumer?
>> >
>> > > PIP briefly talks about it but I will update the PIP with more
>> > > explanation.
>> >
>> > I look forward to seeing more about this design for conflict resolution.
>> >
>> > Thanks,
>> > Michael
>> >
>> >
>> >
>> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dh...@gmail.com>
>> > wrote:
>> > >
>> > > Please find my response inline.
>> > >
>> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <
>> mmarshall@apache.org>
>> > > wrote:
>> > >
>> > > > I think this is a very appropriate direction to take Pulsar's
>> > > > geo-replication. Your proposal is essentially to make the
>> > > > inter-cluster configuration event driven. This increases fault
>> > > > tolerance and better decouples clusters.
>> > > >
>> > > > Thank you for your detailed proposal. After reading through it, I
>> have
>> > > > some questions :)
>> > > >
>> > > > 1. What do you think about using protobuf to define the event
>> > > > protocol? I know we already have a topic policy event stream
>> > > > defined with Java POJOs, but since this feature is specifically
>> > > > designed for egressing cloud providers, ensuring compact data
>> transfer
>> > > > would keep egress costs down. Additionally, protobuf can help make
>> it
>> > > > clear that the schema is strict, should evolve thoughtfully, and
>> > > > should be designed to work between clusters of different versions.
>> > > >
>> > >
>> > >  >>> I don't see a need of protobuf for this particular usecase
>> because
>> > of
>> > > two reasons:
>> > >   >> a. policy changes don't generate huge traffic which could be 1
>> rps
>> > b.
>> > > and it doesn't need performance optimization.
>> > >   >> It should be similar as storing policy in text instead protobuf
>> > which
>> > > doesn't impact footprint size or performance due to limited number of
>> > >  >> update operations and relatively less complexity. I agree that
>> > protobuf
>> > > could be another option but in this case it's not needed. Also, POJO
>> > >  >> can also support schema and versioning.
>> > >
>> > >
>> > >
>> > > >
>> > > > 2. In your view, which tenant/namespace will host
>> > > > `metadataSyncEventTopic`? Will there be several of these topics or
>> is
>> > > > it just hosted in a system tenant/namespace? This question gets back
>> > > > to my questions about system topics on this mailing list last week
>> > [0]. I
>> > > > view this topic as a system topic, so we'd need to make sure that it
>> > > > has the right authorization rules and that it won't be affected by
>> > calls
>> > > > like "clearNamespaceBacklog".
>> > >
>> > >
>> > >   >> It doesn't matter if it's system-topic or not because it's
>> > > configurable and the admin of the system can decide and configure it
>> > > according to the required persistent policy. I would keep the system
>> > topic
>> > > separate because this topic serves a specific purpose with specific
>> > schema,
>> > > replication policy and retention policy.
>> > >
>> > >
>> > >
>> > > >
>> > > > 3. Which broker will host the metadata update publisher? I assume we
>> > > > want the producer to be collocated with the bundle that hosts the
>> > > > event topic. How will this be coordinated?
>> > > >
>> > > >> It's already explained into PIP in section: "Event publisher and
>> > handler"
>> > > >> Every isolated cluster deployed on a separate cloud platform will
>> > have a
>> > > source region and part of replicated clusters for the event topic. The
>> > > Source region will have a broker which will create a failover
>> consumer on
>> > > that topic and a broker with an active consumer will watch the
>> metadata
>> > > changes and publish the changes to the event topic.
>> > >
>> > >
>> > >
>> > > >
>> > > > 4. Why isn't a topic a `ResourceType`? Is this because the topic
>> level
>> > > > policies already have this feature? If so, is there a way to
>> integrate
>> > > > this feature with the existing topic policy feature?
>> > > >
>> > > >> Yes, ResourceType can be extensible to a topic as well.
>> > >
>> > >
>> > >
>> > > >
>> > > > 5. By decentralizing the metadata store, it looks like there is a
>> > > > chance for conflicts due to concurrent updates. How do we handle
>> those
>> > > > conflicts?
>> > > >
>> > > >>  PIP briefly talks about it but I will update the PIP with more
>> > > explanation. MetadataChangeEvent contains source-cluster and updated
>> > time.
>> > > Also, resources Tenant/Namespace will also contain lastUpdatedTime
>> which
>> > > will help to destination clusters to handle stale/duplicate events and
>> > race
>> > > conditions. Also, snapshot-sync an additional task helps all clusters
>> to
>> > be
>> > > synced with each other eventually.
>> > >
>> > >
>> > >
>> > > > I'll also note that I previously proposed a system event topic here
>> > > > [1] and it was proposed again here [2]. Those features were for
>> > > > different use cases, but ultimately looked very similar. In my
>> view, a
>> > > > stream of system events is a very natural feature to expect in a
>> > > > streaming technology. I wonder if there is a way to generalize this
>> > > > feature to fulfill local cluster consumers and geo-replication
>> > > > consumers. Even if this PIP only implements the geo-replication
>> > > > portion of the feature, it'd be good to design it in an extensible
>> > fashion.
>> > > >
>> > >  >> I think answer (2) addresses this concern as well.
>> > >
>> > >
>> > >
>> > > > Thanks,
>> > > > Michael
>> > > >
>> > > > [0]
>> https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
>> > > > [1]
>> https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
>> > > > [2]
>> https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
>> > > >
>> > > >
>> > > >
>> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
>> rdhabalia@apache.org>
>> > > > wrote:
>> > > > >
>> > > > > Hi,
>> > > > >
>> > > > > I would like to start a discussion about PIP-136: Sync Pulsar
>> > policies
>> > > > > across multiple clouds.
>> > > > >
>> > > > > PIP documentation: https://github.com/apache/pulsar/issues/13728
>> > > > >
>> > > > > *Motivation*
>> > > > > Apache Pulsar is a cloud-native, distributed messaging framework
>> > which
>> > > > > natively provides geo-replication. Many organizations deploy
>> pulsar
>> > > > > instances on-prem and on multiple different cloud providers and at
>> > the
>> > > > same
>> > > > > time they would like to enable replication between multiple
>> clusters
>> > > > > deployed in different cloud providers. Pulsar already provides
>> > various
>> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to
>> > > > fulfill
>> > > > > security requirements when brokers are deployed on different
>> security
>> > > > zones
>> > > > > connected with each other. However, sometimes it's not possible to
>> > share
>> > > > > metadata-store (global zookeeper) between pulsar clusters
>> deployed on
>> > > > > separate cloud provider platforms, and synchronizing configuration
>> > > > metadata
>> > > > > (policies) can be a critical path to share tenant/namespace/topic
>> > > > policies
>> > > > > between clusters and administrate pulsar policies uniformly across
>> > all
>> > > > > clusters. Therefore, we need a mechanism to sync configuration
>> > metadata
>> > > > > between clusters deployed on the different cloud platforms.
>> > > > >
>> > > > > *Sync Pulsar policies across multiple clouds*
>> > > > > https://github.com/apache/pulsar/issues/13728
>> > > > > Prototype git-hub-link
>> > > > > <
>> > > >
>> >
>> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
>> > > > >
>> > > > > Thanks,
>> > > > > Rajan
>> > > >
>> >
>>
>

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <dh...@gmail.com>]

>> How do we designate the host broker? Is it manual? How does it work when
the host broker is removed from the cluster?
No, it will not be manual but as I explained earlier a broker which has a
failover consumer to consume remote events will be the publisher for
metadata update. If that broker is removed then a new failover
consumer/broker will be selected for the same.

>> I look forward to seeing more about this design for conflict resolution.
Sure, I have updated PIP to handle such race condition:
https://github.com/apache/pulsar/issues/13728


>> (1) scenarios where the Pulsar cluster operators and tenant admins  are
different entities and tenants can be malicious, or more probably, write
bad code that will produce malicious outcomes.
I agree, Pulsar should have provision to prevent such scenarios where
changes from one tenant in a cluster can impact other clusters. This PIP
considers the tenant/admin will be the same at both the ends but that can
not be true in all cases. We can add an enhancement later or we can create
a separate PIP to start discussion with the possible solutions.

Thanks,
Rajan

On Thu, Feb 3, 2022 at 9:59 AM Joe F <jo...@gmail.com> wrote:

> >On my first reading, it wasn't clear if there was only one topic
> required for this feature. I now see that the topic is not tied to a
> specific tenant or namespace. As such, we can avoid complicated
> authorization questions by putting the required event topic(s) into a
> "system" tenant and namespace
>
> We should consider complicated questions. We can say why we chose not to
> address it, or why it does not apply. for a particular situation
>
> Many namespace policies are administered by tenants.  As such any tenant
> can load this topic.  Is it possible for one abusive tenant to make your
> system topic dysfunctional?
>
> Pulsar committers should think about
> (1) scenarios where the Pulsar cluster operators and tenant admins  are
> different entities and tenants can be malicious, or more probably, write
> bad code that will produce malicious outcomes.
> (2) whether the changes introduce  additional SPOFs into the cluster.
>
> I don't think this PIP has those issues, but  as a matter of practice, I
> would like to see backend/system PIPs consider these questions  and
> explicitly state the conclusions with rationale
>
> Joe
>
>
> On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > Thanks for your responses.
> >
> > > I don't see a need of protobuf for this particular usecase
> >
> > If no one else feels strongly on this point, I am good with using a POJO.
> >
> > > It doesn't matter if it's system-topic or not because it's
> > > configurable and the admin of the system can decide and configure it
> > > according to the required persistent policy.
> >
> > On my first reading, it wasn't clear if there was only one topic
> > required for this feature. I now see that the topic is not tied to a
> > specific tenant or namespace. As such, we can avoid complicated
> > authorization questions by putting the required event topic(s) into a
> > "system" tenant and namespace, by default. The `pulsar/system` tenant
> > and namespace seem appropriate to me.
> >
> > > I would keep the system topic
> > > separate because this topic serves a specific purpose with specific
> > schema,
> > > replication policy and retention policy.
> >
> > I think we need a more formal definition for system topics. This topic
> > is exactly the kind of topic I would call a system topic: its intended
> > producers and consumers are Pulsar components. However, because
> > this feature can live on a topic in a system namespace, we can avoid
> > the classification discussion for this PIP.
> >
> > > Source region will have a broker which will create a failover consumer
> on
> > > that topic and a broker with an active consumer will watch the metadata
> > > changes and publish the changes to the event topic.
> >
> > How do we designate the host broker? Is it manual? How does it work
> > when the host broker is removed from the cluster?
> >
> > If we collocate the active consumer with the broker hosting the event
> > topic, can we skip creating the failover consumer?
> >
> > > PIP briefly talks about it but I will update the PIP with more
> > > explanation.
> >
> > I look forward to seeing more about this design for conflict resolution.
> >
> > Thanks,
> > Michael
> >
> >
> >
> > On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dh...@gmail.com>
> > wrote:
> > >
> > > Please find my response inline.
> > >
> > > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <mmarshall@apache.org
> >
> > > wrote:
> > >
> > > > I think this is a very appropriate direction to take Pulsar's
> > > > geo-replication. Your proposal is essentially to make the
> > > > inter-cluster configuration event driven. This increases fault
> > > > tolerance and better decouples clusters.
> > > >
> > > > Thank you for your detailed proposal. After reading through it, I
> have
> > > > some questions :)
> > > >
> > > > 1. What do you think about using protobuf to define the event
> > > > protocol? I know we already have a topic policy event stream
> > > > defined with Java POJOs, but since this feature is specifically
> > > > designed for egressing cloud providers, ensuring compact data
> transfer
> > > > would keep egress costs down. Additionally, protobuf can help make it
> > > > clear that the schema is strict, should evolve thoughtfully, and
> > > > should be designed to work between clusters of different versions.
> > > >
> > >
> > >  >>> I don't see a need of protobuf for this particular usecase because
> > of
> > > two reasons:
> > >   >> a. policy changes don't generate huge traffic which could be 1 rps
> > b.
> > > and it doesn't need performance optimization.
> > >   >> It should be similar as storing policy in text instead protobuf
> > which
> > > doesn't impact footprint size or performance due to limited number of
> > >  >> update operations and relatively less complexity. I agree that
> > protobuf
> > > could be another option but in this case it's not needed. Also, POJO
> > >  >> can also support schema and versioning.
> > >
> > >
> > >
> > > >
> > > > 2. In your view, which tenant/namespace will host
> > > > `metadataSyncEventTopic`? Will there be several of these topics or is
> > > > it just hosted in a system tenant/namespace? This question gets back
> > > > to my questions about system topics on this mailing list last week
> > [0]. I
> > > > view this topic as a system topic, so we'd need to make sure that it
> > > > has the right authorization rules and that it won't be affected by
> > calls
> > > > like "clearNamespaceBacklog".
> > >
> > >
> > >   >> It doesn't matter if it's system-topic or not because it's
> > > configurable and the admin of the system can decide and configure it
> > > according to the required persistent policy. I would keep the system
> > topic
> > > separate because this topic serves a specific purpose with specific
> > schema,
> > > replication policy and retention policy.
> > >
> > >
> > >
> > > >
> > > > 3. Which broker will host the metadata update publisher? I assume we
> > > > want the producer to be collocated with the bundle that hosts the
> > > > event topic. How will this be coordinated?
> > > >
> > > >> It's already explained into PIP in section: "Event publisher and
> > handler"
> > > >> Every isolated cluster deployed on a separate cloud platform will
> > have a
> > > source region and part of replicated clusters for the event topic. The
> > > Source region will have a broker which will create a failover consumer
> on
> > > that topic and a broker with an active consumer will watch the metadata
> > > changes and publish the changes to the event topic.
> > >
> > >
> > >
> > > >
> > > > 4. Why isn't a topic a `ResourceType`? Is this because the topic
> level
> > > > policies already have this feature? If so, is there a way to
> integrate
> > > > this feature with the existing topic policy feature?
> > > >
> > > >> Yes, ResourceType can be extensible to a topic as well.
> > >
> > >
> > >
> > > >
> > > > 5. By decentralizing the metadata store, it looks like there is a
> > > > chance for conflicts due to concurrent updates. How do we handle
> those
> > > > conflicts?
> > > >
> > > >>  PIP briefly talks about it but I will update the PIP with more
> > > explanation. MetadataChangeEvent contains source-cluster and updated
> > time.
> > > Also, resources Tenant/Namespace will also contain lastUpdatedTime
> which
> > > will help to destination clusters to handle stale/duplicate events and
> > race
> > > conditions. Also, snapshot-sync an additional task helps all clusters
> to
> > be
> > > synced with each other eventually.
> > >
> > >
> > >
> > > > I'll also note that I previously proposed a system event topic here
> > > > [1] and it was proposed again here [2]. Those features were for
> > > > different use cases, but ultimately looked very similar. In my view,
> a
> > > > stream of system events is a very natural feature to expect in a
> > > > streaming technology. I wonder if there is a way to generalize this
> > > > feature to fulfill local cluster consumers and geo-replication
> > > > consumers. Even if this PIP only implements the geo-replication
> > > > portion of the feature, it'd be good to design it in an extensible
> > fashion.
> > > >
> > >  >> I think answer (2) addresses this concern as well.
> > >
> > >
> > >
> > > > Thanks,
> > > > Michael
> > > >
> > > > [0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > > [1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > > [2] https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > > >
> > > >
> > > >
> > > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <
> rdhabalia@apache.org>
> > > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I would like to start a discussion about PIP-136: Sync Pulsar
> > policies
> > > > > across multiple clouds.
> > > > >
> > > > > PIP documentation: https://github.com/apache/pulsar/issues/13728
> > > > >
> > > > > *Motivation*
> > > > > Apache Pulsar is a cloud-native, distributed messaging framework
> > which
> > > > > natively provides geo-replication. Many organizations deploy pulsar
> > > > > instances on-prem and on multiple different cloud providers and at
> > the
> > > > same
> > > > > time they would like to enable replication between multiple
> clusters
> > > > > deployed in different cloud providers. Pulsar already provides
> > various
> > > > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to
> > > > fulfill
> > > > > security requirements when brokers are deployed on different
> security
> > > > zones
> > > > > connected with each other. However, sometimes it's not possible to
> > share
> > > > > metadata-store (global zookeeper) between pulsar clusters deployed
> on
> > > > > separate cloud provider platforms, and synchronizing configuration
> > > > metadata
> > > > > (policies) can be a critical path to share tenant/namespace/topic
> > > > policies
> > > > > between clusters and administrate pulsar policies uniformly across
> > all
> > > > > clusters. Therefore, we need a mechanism to sync configuration
> > metadata
> > > > > between clusters deployed on the different cloud platforms.
> > > > >
> > > > > *Sync Pulsar policies across multiple clouds*
> > > > > https://github.com/apache/pulsar/issues/13728
> > > > > Prototype git-hub-link
> > > > > <
> > > >
> >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > > >
> > > > > Thanks,
> > > > > Rajan
> > > >
> >
>

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Joe F <jo...@gmail.com>]

>On my first reading, it wasn't clear if there was only one topic
required for this feature. I now see that the topic is not tied to a
specific tenant or namespace. As such, we can avoid complicated
authorization questions by putting the required event topic(s) into a
"system" tenant and namespace

We should consider complicated questions. We can say why we chose not to
address it, or why it does not apply. for a particular situation

Many namespace policies are administered by tenants.  As such any tenant
can load this topic.  Is it possible for one abusive tenant to make your
system topic dysfunctional?

Pulsar committers should think about
(1) scenarios where the Pulsar cluster operators and tenant admins  are
different entities and tenants can be malicious, or more probably, write
bad code that will produce malicious outcomes.
(2) whether the changes introduce  additional SPOFs into the cluster.

I don't think this PIP has those issues, but  as a matter of practice, I
would like to see backend/system PIPs consider these questions  and
explicitly state the conclusions with rationale

Joe


On Wed, Feb 2, 2022 at 9:27 PM Michael Marshall <mm...@apache.org>
wrote:

> Thanks for your responses.
>
> > I don't see a need of protobuf for this particular usecase
>
> If no one else feels strongly on this point, I am good with using a POJO.
>
> > It doesn't matter if it's system-topic or not because it's
> > configurable and the admin of the system can decide and configure it
> > according to the required persistent policy.
>
> On my first reading, it wasn't clear if there was only one topic
> required for this feature. I now see that the topic is not tied to a
> specific tenant or namespace. As such, we can avoid complicated
> authorization questions by putting the required event topic(s) into a
> "system" tenant and namespace, by default. The `pulsar/system` tenant
> and namespace seem appropriate to me.
>
> > I would keep the system topic
> > separate because this topic serves a specific purpose with specific
> schema,
> > replication policy and retention policy.
>
> I think we need a more formal definition for system topics. This topic
> is exactly the kind of topic I would call a system topic: its intended
> producers and consumers are Pulsar components. However, because
> this feature can live on a topic in a system namespace, we can avoid
> the classification discussion for this PIP.
>
> > Source region will have a broker which will create a failover consumer on
> > that topic and a broker with an active consumer will watch the metadata
> > changes and publish the changes to the event topic.
>
> How do we designate the host broker? Is it manual? How does it work
> when the host broker is removed from the cluster?
>
> If we collocate the active consumer with the broker hosting the event
> topic, can we skip creating the failover consumer?
>
> > PIP briefly talks about it but I will update the PIP with more
> > explanation.
>
> I look forward to seeing more about this design for conflict resolution.
>
> Thanks,
> Michael
>
>
>
> On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dh...@gmail.com>
> wrote:
> >
> > Please find my response inline.
> >
> > On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <mm...@apache.org>
> > wrote:
> >
> > > I think this is a very appropriate direction to take Pulsar's
> > > geo-replication. Your proposal is essentially to make the
> > > inter-cluster configuration event driven. This increases fault
> > > tolerance and better decouples clusters.
> > >
> > > Thank you for your detailed proposal. After reading through it, I have
> > > some questions :)
> > >
> > > 1. What do you think about using protobuf to define the event
> > > protocol? I know we already have a topic policy event stream
> > > defined with Java POJOs, but since this feature is specifically
> > > designed for egressing cloud providers, ensuring compact data transfer
> > > would keep egress costs down. Additionally, protobuf can help make it
> > > clear that the schema is strict, should evolve thoughtfully, and
> > > should be designed to work between clusters of different versions.
> > >
> >
> >  >>> I don't see a need of protobuf for this particular usecase because
> of
> > two reasons:
> >   >> a. policy changes don't generate huge traffic which could be 1 rps
> b.
> > and it doesn't need performance optimization.
> >   >> It should be similar as storing policy in text instead protobuf
> which
> > doesn't impact footprint size or performance due to limited number of
> >  >> update operations and relatively less complexity. I agree that
> protobuf
> > could be another option but in this case it's not needed. Also, POJO
> >  >> can also support schema and versioning.
> >
> >
> >
> > >
> > > 2. In your view, which tenant/namespace will host
> > > `metadataSyncEventTopic`? Will there be several of these topics or is
> > > it just hosted in a system tenant/namespace? This question gets back
> > > to my questions about system topics on this mailing list last week
> [0]. I
> > > view this topic as a system topic, so we'd need to make sure that it
> > > has the right authorization rules and that it won't be affected by
> calls
> > > like "clearNamespaceBacklog".
> >
> >
> >   >> It doesn't matter if it's system-topic or not because it's
> > configurable and the admin of the system can decide and configure it
> > according to the required persistent policy. I would keep the system
> topic
> > separate because this topic serves a specific purpose with specific
> schema,
> > replication policy and retention policy.
> >
> >
> >
> > >
> > > 3. Which broker will host the metadata update publisher? I assume we
> > > want the producer to be collocated with the bundle that hosts the
> > > event topic. How will this be coordinated?
> > >
> > >> It's already explained into PIP in section: "Event publisher and
> handler"
> > >> Every isolated cluster deployed on a separate cloud platform will
> have a
> > source region and part of replicated clusters for the event topic. The
> > Source region will have a broker which will create a failover consumer on
> > that topic and a broker with an active consumer will watch the metadata
> > changes and publish the changes to the event topic.
> >
> >
> >
> > >
> > > 4. Why isn't a topic a `ResourceType`? Is this because the topic level
> > > policies already have this feature? If so, is there a way to integrate
> > > this feature with the existing topic policy feature?
> > >
> > >> Yes, ResourceType can be extensible to a topic as well.
> >
> >
> >
> > >
> > > 5. By decentralizing the metadata store, it looks like there is a
> > > chance for conflicts due to concurrent updates. How do we handle those
> > > conflicts?
> > >
> > >>  PIP briefly talks about it but I will update the PIP with more
> > explanation. MetadataChangeEvent contains source-cluster and updated
> time.
> > Also, resources Tenant/Namespace will also contain lastUpdatedTime which
> > will help to destination clusters to handle stale/duplicate events and
> race
> > conditions. Also, snapshot-sync an additional task helps all clusters to
> be
> > synced with each other eventually.
> >
> >
> >
> > > I'll also note that I previously proposed a system event topic here
> > > [1] and it was proposed again here [2]. Those features were for
> > > different use cases, but ultimately looked very similar. In my view, a
> > > stream of system events is a very natural feature to expect in a
> > > streaming technology. I wonder if there is a way to generalize this
> > > feature to fulfill local cluster consumers and geo-replication
> > > consumers. Even if this PIP only implements the geo-replication
> > > portion of the feature, it'd be good to design it in an extensible
> fashion.
> > >
> >  >> I think answer (2) addresses this concern as well.
> >
> >
> >
> > > Thanks,
> > > Michael
> > >
> > > [0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > > [1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > > [2] https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> > >
> > >
> > >
> > > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <rd...@apache.org>
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > I would like to start a discussion about PIP-136: Sync Pulsar
> policies
> > > > across multiple clouds.
> > > >
> > > > PIP documentation: https://github.com/apache/pulsar/issues/13728
> > > >
> > > > *Motivation*
> > > > Apache Pulsar is a cloud-native, distributed messaging framework
> which
> > > > natively provides geo-replication. Many organizations deploy pulsar
> > > > instances on-prem and on multiple different cloud providers and at
> the
> > > same
> > > > time they would like to enable replication between multiple clusters
> > > > deployed in different cloud providers. Pulsar already provides
> various
> > > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to
> > > fulfill
> > > > security requirements when brokers are deployed on different security
> > > zones
> > > > connected with each other. However, sometimes it's not possible to
> share
> > > > metadata-store (global zookeeper) between pulsar clusters deployed on
> > > > separate cloud provider platforms, and synchronizing configuration
> > > metadata
> > > > (policies) can be a critical path to share tenant/namespace/topic
> > > policies
> > > > between clusters and administrate pulsar policies uniformly across
> all
> > > > clusters. Therefore, we need a mechanism to sync configuration
> metadata
> > > > between clusters deployed on the different cloud platforms.
> > > >
> > > > *Sync Pulsar policies across multiple clouds*
> > > > https://github.com/apache/pulsar/issues/13728
> > > > Prototype git-hub-link
> > > > <
> > >
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > > >
> > > > Thanks,
> > > > Rajan
> > >
>

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Michael Marshall <mm...@apache.org>]

Thanks for your responses.

> I don't see a need of protobuf for this particular usecase

If no one else feels strongly on this point, I am good with using a POJO.

> It doesn't matter if it's system-topic or not because it's
> configurable and the admin of the system can decide and configure it
> according to the required persistent policy.

On my first reading, it wasn't clear if there was only one topic
required for this feature. I now see that the topic is not tied to a
specific tenant or namespace. As such, we can avoid complicated
authorization questions by putting the required event topic(s) into a
"system" tenant and namespace, by default. The `pulsar/system` tenant
and namespace seem appropriate to me.

> I would keep the system topic
> separate because this topic serves a specific purpose with specific schema,
> replication policy and retention policy.

I think we need a more formal definition for system topics. This topic
is exactly the kind of topic I would call a system topic: its intended
producers and consumers are Pulsar components. However, because
this feature can live on a topic in a system namespace, we can avoid
the classification discussion for this PIP.

> Source region will have a broker which will create a failover consumer on
> that topic and a broker with an active consumer will watch the metadata
> changes and publish the changes to the event topic.

How do we designate the host broker? Is it manual? How does it work
when the host broker is removed from the cluster?

If we collocate the active consumer with the broker hosting the event
topic, can we skip creating the failover consumer?

> PIP briefly talks about it but I will update the PIP with more
> explanation.

I look forward to seeing more about this design for conflict resolution.

Thanks,
Michael



On Tue, Feb 1, 2022 at 3:01 AM Rajan Dhabalia <dh...@gmail.com> wrote:
>
> Please find my response inline.
>
> On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <mm...@apache.org>
> wrote:
>
> > I think this is a very appropriate direction to take Pulsar's
> > geo-replication. Your proposal is essentially to make the
> > inter-cluster configuration event driven. This increases fault
> > tolerance and better decouples clusters.
> >
> > Thank you for your detailed proposal. After reading through it, I have
> > some questions :)
> >
> > 1. What do you think about using protobuf to define the event
> > protocol? I know we already have a topic policy event stream
> > defined with Java POJOs, but since this feature is specifically
> > designed for egressing cloud providers, ensuring compact data transfer
> > would keep egress costs down. Additionally, protobuf can help make it
> > clear that the schema is strict, should evolve thoughtfully, and
> > should be designed to work between clusters of different versions.
> >
>
>  >>> I don't see a need of protobuf for this particular usecase because of
> two reasons:
>   >> a. policy changes don't generate huge traffic which could be 1 rps b.
> and it doesn't need performance optimization.
>   >> It should be similar as storing policy in text instead protobuf which
> doesn't impact footprint size or performance due to limited number of
>  >> update operations and relatively less complexity. I agree that protobuf
> could be another option but in this case it's not needed. Also, POJO
>  >> can also support schema and versioning.
>
>
>
> >
> > 2. In your view, which tenant/namespace will host
> > `metadataSyncEventTopic`? Will there be several of these topics or is
> > it just hosted in a system tenant/namespace? This question gets back
> > to my questions about system topics on this mailing list last week [0]. I
> > view this topic as a system topic, so we'd need to make sure that it
> > has the right authorization rules and that it won't be affected by calls
> > like "clearNamespaceBacklog".
>
>
>   >> It doesn't matter if it's system-topic or not because it's
> configurable and the admin of the system can decide and configure it
> according to the required persistent policy. I would keep the system topic
> separate because this topic serves a specific purpose with specific schema,
> replication policy and retention policy.
>
>
>
> >
> > 3. Which broker will host the metadata update publisher? I assume we
> > want the producer to be collocated with the bundle that hosts the
> > event topic. How will this be coordinated?
> >
> >> It's already explained into PIP in section: "Event publisher and handler"
> >> Every isolated cluster deployed on a separate cloud platform will have a
> source region and part of replicated clusters for the event topic. The
> Source region will have a broker which will create a failover consumer on
> that topic and a broker with an active consumer will watch the metadata
> changes and publish the changes to the event topic.
>
>
>
> >
> > 4. Why isn't a topic a `ResourceType`? Is this because the topic level
> > policies already have this feature? If so, is there a way to integrate
> > this feature with the existing topic policy feature?
> >
> >> Yes, ResourceType can be extensible to a topic as well.
>
>
>
> >
> > 5. By decentralizing the metadata store, it looks like there is a
> > chance for conflicts due to concurrent updates. How do we handle those
> > conflicts?
> >
> >>  PIP briefly talks about it but I will update the PIP with more
> explanation. MetadataChangeEvent contains source-cluster and updated time.
> Also, resources Tenant/Namespace will also contain lastUpdatedTime which
> will help to destination clusters to handle stale/duplicate events and race
> conditions. Also, snapshot-sync an additional task helps all clusters to be
> synced with each other eventually.
>
>
>
> > I'll also note that I previously proposed a system event topic here
> > [1] and it was proposed again here [2]. Those features were for
> > different use cases, but ultimately looked very similar. In my view, a
> > stream of system events is a very natural feature to expect in a
> > streaming technology. I wonder if there is a way to generalize this
> > feature to fulfill local cluster consumers and geo-replication
> > consumers. Even if this PIP only implements the geo-replication
> > portion of the feature, it'd be good to design it in an extensible fashion.
> >
>  >> I think answer (2) addresses this concern as well.
>
>
>
> > Thanks,
> > Michael
> >
> > [0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> > [1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> > [2] https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
> >
> >
> >
> > On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <rd...@apache.org>
> > wrote:
> > >
> > > Hi,
> > >
> > > I would like to start a discussion about PIP-136: Sync Pulsar policies
> > > across multiple clouds.
> > >
> > > PIP documentation: https://github.com/apache/pulsar/issues/13728
> > >
> > > *Motivation*
> > > Apache Pulsar is a cloud-native, distributed messaging framework which
> > > natively provides geo-replication. Many organizations deploy pulsar
> > > instances on-prem and on multiple different cloud providers and at the
> > same
> > > time they would like to enable replication between multiple clusters
> > > deployed in different cloud providers. Pulsar already provides various
> > > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to
> > fulfill
> > > security requirements when brokers are deployed on different security
> > zones
> > > connected with each other. However, sometimes it's not possible to share
> > > metadata-store (global zookeeper) between pulsar clusters deployed on
> > > separate cloud provider platforms, and synchronizing configuration
> > metadata
> > > (policies) can be a critical path to share tenant/namespace/topic
> > policies
> > > between clusters and administrate pulsar policies uniformly across all
> > > clusters. Therefore, we need a mechanism to sync configuration metadata
> > > between clusters deployed on the different cloud platforms.
> > >
> > > *Sync Pulsar policies across multiple clouds*
> > > https://github.com/apache/pulsar/issues/13728
> > > Prototype git-hub-link
> > > <
> > https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> > >
> > > Thanks,
> > > Rajan
> >

Re: [DISCUSS] PIP-136: Sync Pulsar policies across multiple clouds

[Rajan Dhabalia <dh...@gmail.com>]

Please find my response inline.

On Mon, Jan 31, 2022 at 9:17 PM Michael Marshall <mm...@apache.org>
wrote:

> I think this is a very appropriate direction to take Pulsar's
> geo-replication. Your proposal is essentially to make the
> inter-cluster configuration event driven. This increases fault
> tolerance and better decouples clusters.
>
> Thank you for your detailed proposal. After reading through it, I have
> some questions :)
>
> 1. What do you think about using protobuf to define the event
> protocol? I know we already have a topic policy event stream
> defined with Java POJOs, but since this feature is specifically
> designed for egressing cloud providers, ensuring compact data transfer
> would keep egress costs down. Additionally, protobuf can help make it
> clear that the schema is strict, should evolve thoughtfully, and
> should be designed to work between clusters of different versions.
>

 >>> I don't see a need of protobuf for this particular usecase because of
two reasons:
  >> a. policy changes don't generate huge traffic which could be 1 rps b.
and it doesn't need performance optimization.
  >> It should be similar as storing policy in text instead protobuf which
doesn't impact footprint size or performance due to limited number of
 >> update operations and relatively less complexity. I agree that protobuf
could be another option but in this case it's not needed. Also, POJO
 >> can also support schema and versioning.



>
> 2. In your view, which tenant/namespace will host
> `metadataSyncEventTopic`? Will there be several of these topics or is
> it just hosted in a system tenant/namespace? This question gets back
> to my questions about system topics on this mailing list last week [0]. I
> view this topic as a system topic, so we'd need to make sure that it
> has the right authorization rules and that it won't be affected by calls
> like "clearNamespaceBacklog".


  >> It doesn't matter if it's system-topic or not because it's
configurable and the admin of the system can decide and configure it
according to the required persistent policy. I would keep the system topic
separate because this topic serves a specific purpose with specific schema,
replication policy and retention policy.



>
> 3. Which broker will host the metadata update publisher? I assume we
> want the producer to be collocated with the bundle that hosts the
> event topic. How will this be coordinated?
>
>> It's already explained into PIP in section: "Event publisher and handler"
>> Every isolated cluster deployed on a separate cloud platform will have a
source region and part of replicated clusters for the event topic. The
Source region will have a broker which will create a failover consumer on
that topic and a broker with an active consumer will watch the metadata
changes and publish the changes to the event topic.



>
> 4. Why isn't a topic a `ResourceType`? Is this because the topic level
> policies already have this feature? If so, is there a way to integrate
> this feature with the existing topic policy feature?
>
>> Yes, ResourceType can be extensible to a topic as well.



>
> 5. By decentralizing the metadata store, it looks like there is a
> chance for conflicts due to concurrent updates. How do we handle those
> conflicts?
>
>>  PIP briefly talks about it but I will update the PIP with more
explanation. MetadataChangeEvent contains source-cluster and updated time.
Also, resources Tenant/Namespace will also contain lastUpdatedTime which
will help to destination clusters to handle stale/duplicate events and race
conditions. Also, snapshot-sync an additional task helps all clusters to be
synced with each other eventually.



> I'll also note that I previously proposed a system event topic here
> [1] and it was proposed again here [2]. Those features were for
> different use cases, but ultimately looked very similar. In my view, a
> stream of system events is a very natural feature to expect in a
> streaming technology. I wonder if there is a way to generalize this
> feature to fulfill local cluster consumers and geo-replication
> consumers. Even if this PIP only implements the geo-replication
> portion of the feature, it'd be good to design it in an extensible fashion.
>
 >> I think answer (2) addresses this concern as well.



> Thanks,
> Michael
>
> [0] https://lists.apache.org/thread/pj4n4wzm3do8nkc52l7g7obh0sktzm17
> [1] https://lists.apache.org/thread/h4cbvwjdomktsq2jo66x5qpvhdrqk871
> [2] https://lists.apache.org/thread/0xkg0gpsobp0dbgb6tp9xq097lpm65bx
>
>
>
> On Sun, Jan 30, 2022 at 10:33 PM Rajan Dhabalia <rd...@apache.org>
> wrote:
> >
> > Hi,
> >
> > I would like to start a discussion about PIP-136: Sync Pulsar policies
> > across multiple clouds.
> >
> > PIP documentation: https://github.com/apache/pulsar/issues/13728
> >
> > *Motivation*
> > Apache Pulsar is a cloud-native, distributed messaging framework which
> > natively provides geo-replication. Many organizations deploy pulsar
> > instances on-prem and on multiple different cloud providers and at the
> same
> > time they would like to enable replication between multiple clusters
> > deployed in different cloud providers. Pulsar already provides various
> > proxy options (Pulsar proxy/ enterprise proxy solutions on SNI) to
> fulfill
> > security requirements when brokers are deployed on different security
> zones
> > connected with each other. However, sometimes it's not possible to share
> > metadata-store (global zookeeper) between pulsar clusters deployed on
> > separate cloud provider platforms, and synchronizing configuration
> metadata
> > (policies) can be a critical path to share tenant/namespace/topic
> policies
> > between clusters and administrate pulsar policies uniformly across all
> > clusters. Therefore, we need a mechanism to sync configuration metadata
> > between clusters deployed on the different cloud platforms.
> >
> > *Sync Pulsar policies across multiple clouds*
> > https://github.com/apache/pulsar/issues/13728
> > Prototype git-hub-link
> > <
> https://github.com/rdhabalia/pulsar/commit/e59803b942918076ce6376b50b35ca827a49bcf6
> >
> > Thanks,
> > Rajan
>