You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@pulsar.apache.org by Zhengxin Cai <ca...@gmail.com> on 2019/04/08 04:11:06 UTC

[DISCUSS] PIP 35: Provide a batch lookup api for partitioned api

Hi all:

As this github issue <https://github.com/apache/pulsar/issues/1088> brings
up, for a partitioned topic, client has to send LookUp request for each
partition which would result in quite a lot of topic lookup requests when a
topic has many partitions. Here I'd like to propose to add a batch lookup
api and make lookup easier for partitioned topic.

We could create a new BatchLookUp api, like getBrokers. It can take a list
of topic as input, where each topicname has to be either a non-partitioned
topic or a partition of a partitioned topic and it’ll return
Map<TopicName, Pair<InetSocketAddress,
InetSocketAddress>> where it’s a map of input topic to corresponding
address.


Here's a google doc contains more information and some proposed changes:
https://docs.google.com/document/d/1yJ6GvtrdojY7EjQ4ZJK8dzX1M2i185Nf-dpwvsPWb1w/edit.


Any ideas and suggestions are welcome.


Thanks,

Re: [DISCUSS] PIP 35: Provide a batch lookup api for partitioned api

Posted by Sijie Guo <gu...@gmail.com>.

Thanks Zhengxin! This will be a great improvement on handling large number
of partitions.

I have moved your PIP to pulsar wiki:
https://github.com/apache/pulsar/wiki/PIP-35:-Improve-topic-lookup-for-topics-that-have-high-number-of-partitions

The proposal looks good to me. Looking forward to the implementation!

- Sijie

On Mon, Apr 8, 2019 at 12:19 PM Zhengxin Cai <ca...@gmail.com> wrote:

> Hi all:
>
> As this github issue <https://github.com/apache/pulsar/issues/1088> brings
> up, for a partitioned topic, client has to send LookUp request for each
> partition which would result in quite a lot of topic lookup requests when a
> topic has many partitions. Here I'd like to propose to add a batch lookup
> api and make lookup easier for partitioned topic.
>
> We could create a new BatchLookUp api, like getBrokers. It can take a list
> of topic as input, where each topicname has to be either a non-partitioned
> topic or a partition of a partitioned topic and it’ll return
> Map<TopicName, Pair<InetSocketAddress,
> InetSocketAddress>> where it’s a map of input topic to corresponding
> address.
>
>
> Here's a google doc contains more information and some proposed changes:
>
> https://docs.google.com/document/d/1yJ6GvtrdojY7EjQ4ZJK8dzX1M2i185Nf-dpwvsPWb1w/edit
> .
>
>
> Any ideas and suggestions are welcome.
>
>
> Thanks,
>