Re: [PIP-48] Hierarchical admin api

[Matteo Merli <mm...@apache.org>]

On Thu, Oct 31, 2019 at 4:12 AM Sijie Guo <gu...@gmail.com> wrote:
>
> Hi Matteo,
>
> I think the main proposal that Florentin and Steven proposed is to define a
> `scope` for each restful api.

I get the "scope" part, though it comes out confusing in proposal since there
are multiple statements either not accurate or without precise context and
definition. That's why I've asked for clarifications.

> The scope indicates where a client should send the http request. The
> current restful APIs can be categorized into three scopes:
>
> - broker: send to this broker.
> - cluster: send to any broker in this cluster.
> -  instance (constellation as proposed here): send to any broker at any
> cluster in the instance.
>
> We also encountered some of the pain points that Florentin and Steven
> encountered when developing Pulsar Manager.

Yes, there are of course difference scopes for different requests and I agree
it's not always easy to figure out. Additionally, in some cases a
broker will proxy
a request to the correct broker, while in some other cases the caller
is expected
to issue it to that broker.


Some of the scopes in the proposal are a bit off though:

> v4/clusters/:cluster-id/brokers --> manage brokers of the cluster-idcluster, broker

Sure, that's correct and more clear that the current API.

> v4/tenants/:tenant-id/namespaces/:namespace-id/topics

Topic listing and related operations are actually scoped to a
particular cluster.
Though it might not be a good idea to ask for cluster name here because a "user"
(as opposed to an "admin") is not supposed to know the cluster name in most
cases.

Also I don't see a structural difference from the current API handler:
v2/namespaces/:tenant-id/:namespace-id/topics



In my view, if we're going to bump to a `v4` API, this should be the
point in time to do
a thorough revisit of all the API, since many handlers have been organically
added throughout the time with different degrees of coherence.

There should be a focus on ensuring consistency and uniformity of the naming
and parameters. Also, many endpoint should probably be collapsed into a
smaller set (eg: all the policies updates for a namespace).


Finally, I'd leave the original authors to clarify rather than guess
on their intentions.

--
Matteo Merli
<mm...@apache.org>


>
> Because the lack of scope in restful API, people makes mistakes when send
> the http request. So we ended adding a backend
> in the Pulsar Manger and inject scope information by ourselves and use it
> to get the right service url to send the http requests.
>
> I wish we can look into resolve this problem in a newer version. The
> proposed PIP tries to organize the restful apis in a better way
>  to make people easier figure out where they should send the http request.



>
> Whether to rename "instance" to "constellation" can probably be moved to a
> separate PIP. We can discuss it in a separate thread.
>
> - Sijie
>
>
>
> On Thu, Oct 31, 2019 at 12:21 AM Matteo Merli <ma...@gmail.com>
> wrote:
>
> > Hi Florentin,
> >
> > I have bit of trouble understanding the motivation and scope of this
> > proposal. There seems to be a mix of multiple different items.
> >
> >  >  Also, Pulsar evoled to the ability to manage multiple clusters.
> >
> > Pulsar had the ability of managing multiple clusters from day 0 (or I
> > should say day -1200): multiple clusters in a single logical
> > "instance" spanning across multiple geo-distributed data centers.
> >
> > It has not "evolved" into that.
> >
> > > For example to administrate a namespace, we used the following route
> > > v2/namespaces/:tenant-id/:namespace-id. This could be confusing,
> > > because we intend to manipulate a namespace under a tenant scope,
> > > which still requires here to give the tenant identifier in addition of
> > the namespace identifier.
> >
> > A namespace *is* identified by TENANT_ID/NAMESPACE_ID. The REST API
> > just reflects that fact.
> >
> > A namespace only exist in the context of a tenant. Through ACLs, one
> > can decide to give permission to other tenants to
> > produce/consume though that doesn't change the fundamental property
> > that a namespace is always scoped inside a tenant.
> >
> >
> > > offer a more hierarchical routing approach reflecting the Pulsar semantic
> >
> > It's not clear to me what hierarchical here means or in which way the
> > current API is not hierarchical.
> >
> > > proposes to officially name an ensemble of Pulsar cluster : a
> > Constellation
> >
> > I'm not sure anyone can immediately grasp what a "Constellation" is in
> > this context. We've been using "instance" in the
> > past and there was a proposal to use "Federation". I'm open for better
> > proposal though I think we should strive to use a
> > term that is commonly used in the context of a "group of logically
> > related clusters".
> >
> > Also it would have to be clearldefined what an "an ensemble of Pulsar
> > clusters" is.
> >
> > > to simplify the user experience for :
> > > Topic management between persistent and non-persistent.
> >
> > This was not changed on REST API for compatibility (just on JAVA admin
> > and CLI), though I I think that fixing it should not require
> > any breaking changes, just adding a new handler and deprecating the
> > current ones.
> >
> > > cluster management inside a Pulsar Constellation
> >
> > What does this means? It's already possible to manage the metadata of
> > the clusters since the configuration metadata is stored in the global
> > zookeeper instance.
> >
> >
> >
> > >>> v4/metricsbroker metrics (both broker and worker) in prometheus format
> >
> > Prometheus metrics should be exposed at /metrics only
> >
> > >>> v4/broker/statusretrieve broker statusbroker
> >
> > What's the broker status? And why cannot a new handler just be added
> > to the current v2 API?
> >
> >
> > >>> v4/clustersList and create cluster instances constellation
> >
> > How's that different from admin/v2/clusters?
> >
> >
> >
> >
> > On Wed, Oct 30, 2019 at 8:15 AM Florentin Dubois
> > <fl...@corp.ovh.com> wrote:
> > >
> > > Hello,
> > >
> > >
> > > We (Steven and I) have written a draft of pulsar improvement proposal
> > about the admin api.
> > >
> > >
> > > We would like to get some feedback/review about it :D
> > >
> > >
> > > The pip is here:
> > https://gist.github.com/FlorentinDUBOIS/4f98c0f71bf6514309b0414290b48e6a
> > >
> > >
> > > Feel free to contact me on slack or on this mailing list, if you have
> > any questions ;).
> > >
> > >
> > > Have a nice day!
> > >
> > >
> > > ---
> > >
> > > Florentin Dubois - OVH Group
> > >
> > > IO - Tech Lead
> > >
> > > florentin.dubois@corp.ovh.com
> > >
> > > +33 6 58 37 43 83
> > >
> >