Re: Proposal on policies management at tenant level

[Sijie Guo <gu...@gmail.com>]

Hi Alexandre,

Thank you for bringing this topic up. I copied your content to Pulsar wiki
page as PIP-51:
https://github.com/apache/pulsar/wiki/PIP-51%3A-Tenant-policy-support

This proposal is very similar as PIP-39, where we are introducing a
topic-level policy management via namespace change events.

It is in general good to have a tenant level policy. But just keep things
in mind, most of the properties in a policy are applied to a *topic*. That
means if you specify a quota in a namespace policy, it is only used for
limiting the quota of a topic (partition) under that namespace. It just
means all the topics of a namespace has the same constraint as defined in
the policy.

If we are introducing a tenant policy, I would suggest making the tenant
policy as a default policy that applies to all the topics underneath it. If
we want to add a policy for aggregated metrics (or resource usage), we
should first discuss how to implement aggregated resource control over a
set of topic (partitions).

- Sijie

On Thu, Nov 28, 2019 at 8:14 AM Alexandre DUVAL <
alexandre.duval@clever-cloud.com> wrote:

> Hi,
>
> Here is a potential proposal (PIP) on policies management at tenant level:
>
> ## Motivation
>
> Pulsar quotas, retentions and other policies are defined on namespaces
> level and only enforced at topic level. It would be great to have global
> policies on tenant level which can be overriden if policies are lower in
> namespaces level.
>
> The main goal is to provide a tenant for external users and a the way to
> define namespaces and so on with tenant level policies defined by the
> tenant provider.
>
> ## Proposed changes
>
> The tenant's adminRoles property would be used to define the tenant's
> level policies. Then we will need to add a tenant userRoles which should be
> able to create namespaces and everything in the tenant that does not exceed
> the tenants level policies.
>
> Then the namespaces policies should inherit from tenant policies with a
> global verifier for all namespaces to not exceed tenant policies.
>
> It wouldn't be too difficult for retention and storage part. We would have
> to just check periodically on the already supplied brokers load report and
> get the overall namespace/tenant quota. If quotas exceed, then block
> producers only (not sure delete exceeded data because of the async block
> part is a good idea).
>
> As said @merlimat, implement rate limiting quotas at tenant or namespace
> level would be more tricky.
>
> So WDYT?
>
> Regards,
>
> Alex.
>
>