You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Enrico Olivelli <eo...@gmail.com> on 2022/09/05 13:50:30 UTC
[DISCUSS] Pulsar 3.0 - dreaming of the ability to rename objects
Hello,
Now that we have started to think about Pulsar 3.0, let me share
another long standing feature that can be implemented mostly only with
such big changes that would deserve a major version bump.
Currently every object managed by Pulsar is directly handled and
referred using a logical, human readable name:
- tenants
- namespaces
- topics
- functions/sinks/sources
This fact is limiting some interesting use case:
- renaming objects (you cannot rename a topic for instance)
- safe delete/recreate (clients may connect to the new version of the
same object, but now it has a different meaning)
- aliasing (giving multiple logical names to the same thing)
Additionally it would be great to have the ability to not have
globally unique tenant names,
but allow that a tenant name is a logical name and its meaning depends
on the user that authenticated: now in a big multi-tenant environment
you cannot have multiple tenants with the same name, so if you are
providing Pulsar as SaaS, all of your customer cannot use the same
"tenant name".
You cannot even have two tenants named "test-tenant" or "demo".
I am not proposing here solutions for this problem, even if I already
have some ideas, but I want to open this discussion and understand if
others on the community would like to work on these kind of problems
Cheers
Enrico
Re: [DISCUSS] Pulsar 3.0 - dreaming of the ability to rename objects
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno mar 6 set 2022 alle ore 07:46 Lari Hotari
<lh...@apache.org> ha scritto:
>
> I have been thinking of the need for supporting renaming tenants, namespaces and topics while doing some initial design for "Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface", thread https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk . (the thread doesn't mention anything about renaming at the moment).
>
> Adding support for renaming would most likely require adding another level of indirection to lookups so that a logical name would be mapped to a globally unique identifier (GUID). There are tradeoffs involved in this change. However, I think it would be useful to at least have full support for renaming topics in a future Pulsar version. I'd also like to add that this type of possible future requirements are yet another reason to let go of "namespace bundles" centric architecture in Pulsar 3.0.
>
> Enrico mentioned the desire to support tenant names that aren't globally unique and instead are contained in the context of the authenticated user. A question about the use case: Would this user have access to multiple tenants so that the user could decide the mapping of logical names to actual tenant names? What is the actual use case and need for this?
One use case I have in mind is when you have a Pulsar cluster shared
by main independent users (customers??), currently each user cannot
use the same "tenant name" used by another user
this is against the goal of giving every user a self-contained and
logically independent workspace.
I know that in the case of Pulsar this would be hard because we don't
have a top level object that is a "container of tenants" and we don't
even have the concept of "user".
In order to support this use case we have to add a new object, like a
"space", and when you connect to Pulsar you set the "space", then
every reference to a tenant is relative to that "space".
Enrico
>
> -Lari
>
> On 2022/09/05 13:50:30 Enrico Olivelli wrote:
> > Hello,
> > Now that we have started to think about Pulsar 3.0, let me share
> > another long standing feature that can be implemented mostly only with
> > such big changes that would deserve a major version bump.
> >
> > Currently every object managed by Pulsar is directly handled and
> > referred using a logical, human readable name:
> > - tenants
> > - namespaces
> > - topics
> > - functions/sinks/sources
> >
> > This fact is limiting some interesting use case:
> > - renaming objects (you cannot rename a topic for instance)
> > - safe delete/recreate (clients may connect to the new version of the
> > same object, but now it has a different meaning)
> > - aliasing (giving multiple logical names to the same thing)
> >
> > Additionally it would be great to have the ability to not have
> > globally unique tenant names,
> > but allow that a tenant name is a logical name and its meaning depends
> > on the user that authenticated: now in a big multi-tenant environment
> > you cannot have multiple tenants with the same name, so if you are
> > providing Pulsar as SaaS, all of your customer cannot use the same
> > "tenant name".
> > You cannot even have two tenants named "test-tenant" or "demo".
> >
> > I am not proposing here solutions for this problem, even if I already
> > have some ideas, but I want to open this discussion and understand if
> > others on the community would like to work on these kind of problems
> >
> >
> > Cheers
> > Enrico
> >
Re: [DISCUSS] Pulsar 3.0 - dreaming of the ability to rename objects
Posted by Lari Hotari <lh...@apache.org>.
I have been thinking of the need for supporting renaming tenants, namespaces and topics while doing some initial design for "Pulsar 3.0 brainstorming: Going beyond PIP-45 Pluggable metadata interface", thread https://lists.apache.org/thread/roohoc9h2gthvmd7t81do4hfjs2gphpk . (the thread doesn't mention anything about renaming at the moment).
Adding support for renaming would most likely require adding another level of indirection to lookups so that a logical name would be mapped to a globally unique identifier (GUID). There are tradeoffs involved in this change. However, I think it would be useful to at least have full support for renaming topics in a future Pulsar version. I'd also like to add that this type of possible future requirements are yet another reason to let go of "namespace bundles" centric architecture in Pulsar 3.0.
Enrico mentioned the desire to support tenant names that aren't globally unique and instead are contained in the context of the authenticated user. A question about the use case: Would this user have access to multiple tenants so that the user could decide the mapping of logical names to actual tenant names? What is the actual use case and need for this?
-Lari
On 2022/09/05 13:50:30 Enrico Olivelli wrote:
> Hello,
> Now that we have started to think about Pulsar 3.0, let me share
> another long standing feature that can be implemented mostly only with
> such big changes that would deserve a major version bump.
>
> Currently every object managed by Pulsar is directly handled and
> referred using a logical, human readable name:
> - tenants
> - namespaces
> - topics
> - functions/sinks/sources
>
> This fact is limiting some interesting use case:
> - renaming objects (you cannot rename a topic for instance)
> - safe delete/recreate (clients may connect to the new version of the
> same object, but now it has a different meaning)
> - aliasing (giving multiple logical names to the same thing)
>
> Additionally it would be great to have the ability to not have
> globally unique tenant names,
> but allow that a tenant name is a logical name and its meaning depends
> on the user that authenticated: now in a big multi-tenant environment
> you cannot have multiple tenants with the same name, so if you are
> providing Pulsar as SaaS, all of your customer cannot use the same
> "tenant name".
> You cannot even have two tenants named "test-tenant" or "demo".
>
> I am not proposing here solutions for this problem, even if I already
> have some ideas, but I want to open this discussion and understand if
> others on the community would like to work on these kind of problems
>
>
> Cheers
> Enrico
>
Re: [DISCUSS] Pulsar 3.0 - dreaming of the ability to rename objects
Posted by Asaf Mesika <as...@gmail.com>.
I don’t have a solution to your problem, but I can share some thoughts I
had when reading your suggestion:
(1) Simplicity
I’m relatively new to Pulsar, and I try to learn it as I go. One thing I
can say for sure: Pulsar is complicated, both user-facing and internals.
Just the other way, I read a thread trying to decide how exactly an
Exclusive / Failover should behave. So even just as a user trying to
understand how to use Pulsar is complicated.
What I’m trying to say is that on one end, your suggestion has more
functionality and more power, yet on the other end, it will, by design,
make Pulsar more complicated.
IMO, Pulsar should focus more on being:
1. Simpler - super easy to use and understand for users.
2. Stable - fewer bugs, fewer failure scenarios.
(2) Pulsar SAAS
One way I can think of implementing your suggestion is to have that naming
in an external layer. So, if the SAAS models customers as accounts, an
account ID 45 would be tenant “account-45”. From REST API and UI
perspective, you just translate it transparently from account-45 to the
real name, and the customer decides to name that tenant.
(3) Aliasing
I think aliases can be implemented today and make it easier to rename in
multiple steps: Add a new name as an alias, and slowly use the new name in
your apps. When done, just rename the topic.
On 5 Sep 2022 at 16:50:30, Enrico Olivelli <eo...@gmail.com> wrote:
> Hello,
> Now that we have started to think about Pulsar 3.0, let me share
> another long standing feature that can be implemented mostly only with
> such big changes that would deserve a major version bump.
>
> Currently every object managed by Pulsar is directly handled and
> referred using a logical, human readable name:
> - tenants
> - namespaces
> - topics
> - functions/sinks/sources
>
> This fact is limiting some interesting use case:
> - renaming objects (you cannot rename a topic for instance)
> - safe delete/recreate (clients may connect to the new version of the
> same object, but now it has a different meaning)
> - aliasing (giving multiple logical names to the same thing)
>
> Additionally it would be great to have the ability to not have
> globally unique tenant names,
> but allow that a tenant name is a logical name and its meaning depends
> on the user that authenticated: now in a big multi-tenant environment
> you cannot have multiple tenants with the same name, so if you are
> providing Pulsar as SaaS, all of your customer cannot use the same
> "tenant name".
> You cannot even have two tenants named "test-tenant" or "demo".
>
> I am not proposing here solutions for this problem, even if I already
> have some ideas, but I want to open this discussion and understand if
> others on the community would like to work on these kind of problems
>
>
> Cheers
> Enrico
>