You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Devin Bost <de...@gmail.com> on 2021/10/25 21:04:50 UTC
List all topics
I've occasionally run into situations where I needed a list of all the
topics in a cluster.
If I want to submit a PR to add something to one of the endpoints to return
a list of all topics in a cluster, what part of the API should that go
under?
e.g. bin/pulsar-admin tenants list-all-topics
bin/pulsar-admin clusters list-all-topics
etc.
Thoughts?
Anyone opposed to this?
Devin G. Bost
Re: List all topics
Posted by "rxl@apache.org" <ra...@gmail.com>.
Hello Enrico and Devin:
Here I have a concern. Assuming that there are many topics in your cluster,
will this request keep failing?
In a production environment, in fact, I see the current phenomenon is that
if there are 20w sub-partitioned topics under a namespace, when you execute
bin/pulsar-admin topics list tenant/ns this command will always fail.
So when we extend this command to the cluster or tenant level, will we
encounter the same problem?
--
Thanks
Xiaolong Ran
Devin Bost <de...@gmail.com> 于2021年10月27日周三 上午12:26写道：
> Enrico,
>
> Great questions. I wrote a bash script, but it's extremely slow due to
> sequential operations that each wait for a response from Pulsar. I could
> write a Python script instead that uses parallel processing or
> multi-threading, but I'm not sure if that's the best approach for this, and
> performance will still be lacking.
>
>
> > - What about partitioned topics? Are we listing partitions?
>
> I'm open to feedback on this.
>
> > - What about permissions? Are we listing only allowed tenants/topics? Or
> is
> this allowed only for super users?
>
> I assumed that it would be executed by a super user, but I'm open to
> suggestions. I don't know how many Pulsar admins routinely grant admin
> permissions to other teams, so is there much of a need to filter the
> results according to permissions?
>
> > - This can be a heavy weight operation, and also the list may be very
> big,
> how do we deal with this problem?
>
> I'm also open to suggestions here.
>
> --
> Devin G. Bost
>
> On Mon, Oct 25, 2021, 3:41 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Devin,
> >
> > Il Lun 25 Ott 2021, 23:05 Devin Bost <de...@gmail.com> ha scritto:
> >
> > > I've occasionally run into situations where I needed a list of all the
> > > topics in a cluster.
> > > If I want to submit a PR to add something to one of the endpoints to
> > return
> > > a list of all topics in a cluster, what part of the API should that go
> > > under?
> > >
> > > e.g. bin/pulsar-admin tenants list-all-topics
> > > bin/pulsar-admin clusters list-all-topics
> > > etc.
> > >
> > > Thoughts?
> > >
> >
> > Some thoughts:
> > - What about partitioned topics? Are we listing partitions?
> > - What about permissions? Are we listing only allowed tenants/topics? Or
> is
> > this allowed only for super users?
> > - This can be a heavy weight operation, and also the list may be very
> big,
> > how do we deal with this problem?
> >
> > Is it possible to create a bash/python/jewellery script that loops on the
> > tenants and gets the list you want?
> >
> > Thanks for sharing your needs and your proposal
> >
> > Enrico
> >
> >
> >
> > > Anyone opposed to this?
> > >
> > > Devin G. Bost
> > >
> >
>
Re: List all topics
Posted by Devin Bost <de...@gmail.com>.
Enrico,
Great questions. I wrote a bash script, but it's extremely slow due to
sequential operations that each wait for a response from Pulsar. I could
write a Python script instead that uses parallel processing or
multi-threading, but I'm not sure if that's the best approach for this, and
performance will still be lacking.
> - What about partitioned topics? Are we listing partitions?
I'm open to feedback on this.
> - What about permissions? Are we listing only allowed tenants/topics? Or
is
this allowed only for super users?
I assumed that it would be executed by a super user, but I'm open to
suggestions. I don't know how many Pulsar admins routinely grant admin
permissions to other teams, so is there much of a need to filter the
results according to permissions?
> - This can be a heavy weight operation, and also the list may be very big,
how do we deal with this problem?
I'm also open to suggestions here.
--
Devin G. Bost
On Mon, Oct 25, 2021, 3:41 PM Enrico Olivelli <eo...@gmail.com> wrote:
> Devin,
>
> Il Lun 25 Ott 2021, 23:05 Devin Bost <de...@gmail.com> ha scritto:
>
> > I've occasionally run into situations where I needed a list of all the
> > topics in a cluster.
> > If I want to submit a PR to add something to one of the endpoints to
> return
> > a list of all topics in a cluster, what part of the API should that go
> > under?
> >
> > e.g. bin/pulsar-admin tenants list-all-topics
> > bin/pulsar-admin clusters list-all-topics
> > etc.
> >
> > Thoughts?
> >
>
> Some thoughts:
> - What about partitioned topics? Are we listing partitions?
> - What about permissions? Are we listing only allowed tenants/topics? Or is
> this allowed only for super users?
> - This can be a heavy weight operation, and also the list may be very big,
> how do we deal with this problem?
>
> Is it possible to create a bash/python/jewellery script that loops on the
> tenants and gets the list you want?
>
> Thanks for sharing your needs and your proposal
>
> Enrico
>
>
>
> > Anyone opposed to this?
> >
> > Devin G. Bost
> >
>
Re: List all topics
Posted by Enrico Olivelli <eo...@gmail.com>.
Devin,
Il Lun 25 Ott 2021, 23:05 Devin Bost <de...@gmail.com> ha scritto:
> I've occasionally run into situations where I needed a list of all the
> topics in a cluster.
> If I want to submit a PR to add something to one of the endpoints to return
> a list of all topics in a cluster, what part of the API should that go
> under?
>
> e.g. bin/pulsar-admin tenants list-all-topics
> bin/pulsar-admin clusters list-all-topics
> etc.
>
> Thoughts?
>
Some thoughts:
- What about partitioned topics? Are we listing partitions?
- What about permissions? Are we listing only allowed tenants/topics? Or is
this allowed only for super users?
- This can be a heavy weight operation, and also the list may be very big,
how do we deal with this problem?
Is it possible to create a bash/python/jewellery script that loops on the
tenants and gets the list you want?
Thanks for sharing your needs and your proposal
Enrico
> Anyone opposed to this?
>
> Devin G. Bost
>