Re: List all topics

["rxl@apache.org" <ra...@gmail.com>]

Hello Enrico and Devin:

Here I have a concern. Assuming that there are many topics in your cluster,
will this request keep failing?

In a production environment, in fact, I see the current phenomenon is that
if there are 20w sub-partitioned topics under a namespace, when you execute
bin/pulsar-admin topics list tenant/ns this command will always fail.

So when we extend this command to the cluster or tenant level, will we
encounter the same problem?

--

Thanks
Xiaolong Ran

Devin Bost <de...@gmail.com> 于2021年10月27日周三 上午12:26写道：

> Enrico,
>
> Great questions. I wrote a bash script, but it's extremely slow due to
> sequential operations that each wait for a response from Pulsar. I could
> write a Python script instead that uses parallel processing or
> multi-threading, but I'm not sure if that's the best approach for this, and
> performance will still be lacking.
>
>
> > - What about partitioned topics? Are we listing partitions?
>
> I'm open to feedback on this.
>
> > - What about permissions? Are we listing only allowed tenants/topics? Or
> is
> this allowed only for super users?
>
> I assumed that it would be executed by a super user, but I'm open to
> suggestions. I don't know how many Pulsar admins routinely grant admin
> permissions to other teams, so is there much of a need to filter the
> results according to permissions?
>
> > - This can be a heavy weight operation, and also the list may be very
> big,
> how do we deal with this problem?
>
> I'm also open to suggestions here.
>
> --
> Devin G. Bost
>
> On Mon, Oct 25, 2021, 3:41 PM Enrico Olivelli <eo...@gmail.com> wrote:
>
> > Devin,
> >
> > Il Lun 25 Ott 2021, 23:05 Devin Bost <de...@gmail.com> ha scritto:
> >
> > > I've occasionally run into situations where I needed a list of all the
> > > topics in a cluster.
> > > If I want to submit a PR to add something to one of the endpoints to
> > return
> > > a list of all topics in a cluster, what part of the API should that go
> > > under?
> > >
> > > e.g. bin/pulsar-admin tenants list-all-topics
> > >        bin/pulsar-admin clusters list-all-topics
> > >        etc.
> > >
> > > Thoughts?
> > >
> >
> > Some thoughts:
> > - What about partitioned topics? Are we listing partitions?
> > - What about permissions? Are we listing only allowed tenants/topics? Or
> is
> > this allowed only for super users?
> > - This can be a heavy weight operation, and also the list may be very
> big,
> > how do we deal with this problem?
> >
> > Is it possible to create a bash/python/jewellery script that loops on the
> > tenants and gets the list you want?
> >
> > Thanks for sharing your needs and your proposal
> >
> > Enrico
> >
> >
> >
> > > Anyone opposed to this?
> > >
> > > Devin G. Bost
> > >
> >
>