[DISCUSS] PIP-39: Namespace Change Events & Topic Policy

[Sijie Guo <gu...@gmail.com>]

Hi all,

I am starting a proposal about adding change events for a namespace to
support topic level policy.

*motivation*

Pulsar is a multi-tenant event streaming system. Administrators can manage
the tenants and namespaces by setting policies at different levels. However
the policies such as retention policy, storage quota policy are only able
to be set at a namespace level. In many use cases, users would like to have
the ability to override a policy at the topic level. But Pulsar doesn’t
support topic level policy. So they ended up mapping topic to namespaces,
which result in using one namespace per topic and it is only not efficient
in managing many namespaces and topics.

I am proposing introducing `__change_events` topic for namespaces. This
`__change_events` topic can be used for implementing topic-level policy in
an efficient way. In this proposal, it uses Pulsar as an event log for
storing namespace change events (the events are topic policy changes in
this case).

This approach has a few benefits:

- It is moving away from using zookeeper to store more metadata and
preventing introducing more traffic to zookeeper.
- It can scale efficiently by using Pulsar as the event log to propagate
policy cache.
- Once the policy changes land in the `__change_events` topic,
administrators can use Pulsar Functions or Presto SQL to query the changes
and audit the whole system.

The detail proposal is in
https://docs.google.com/document/d/1kBiICo3lUvWfeah1YVNSn2Y6qkI6l5-S3WZv5PZcbwk/edit

PIP-39:
https://github.com/apache/pulsar/wiki/PIP-39:-Namespace-Change-Events

Looking forward to any feedback.

Thanks,
Sijie