You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Upendra Yadav <up...@gmail.com> on 2019/08/19 12:21:44 UTC
list of pattern processed topic list
Hi,
I have initialised kafka consumer:
KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<byte[],
byte[]>(consumerConfig);
and subscribed with topic pattern:
consumer.subscribe(Pattern.compile(topicRegex), listener);
now, I'm trying to get the list of topics.
Map<String, List<PartitionInfo>> topicsPartitions = consumer.listTopics();
here topic list is not following given pattern. it is giving all topic list.
Is this expected behaviour? Is there any API to get subscribed topic list
with consumer instance?
or Do I need to write simple prog to process this list with same pattern?
Re: list of pattern processed topic list
Posted by "Matthias J. Sax" <ma...@confluent.io>.
#listTopic() is not related to your subscription/assignment as indicated
by its JavaDocs:
> Get metadata about partitions for all topics that the user is authorized to view. This method will issue a
> remote call to the server.
-Matthias
On 8/19/19 7:02 AM, Kamal Chandraprakash wrote:
> You can use the KafkaConsumer#assignment() method to get all the assigned
> topic-partitions for that consumer instance.
> But, you've to periodically call poll method to get the latest assignment
> which may return records.
> This shortcoming is actively discussed in the below threads.
>
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201908.mbox/<08030a68-3f0b-42db-9b79-dfcd3200cf25%40www.fastmail.com>
> <https://mail-archives.apache.org/mod_mbox/kafka-dev/201908.mbox/%3C08030a68-3f0b-42db-9b79-dfcd3200cf25%40www.fastmail.com%3E>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484
>
>
> On Mon, Aug 19, 2019 at 6:33 PM M. Manna <ma...@gmail.com> wrote:
>
>> Kafka doesn’t have regex subscription. You could get a list of available
>> topic using admin client. From there, you can decide what topics you would
>> like to subscribe to (or process using your logic).
>>
>> Check documentation for AdminClient.listTopics and ListTopicResult holder.
>>
>> I hope this helps.
>>
>> Regards,
>>
>> On Mon, 19 Aug 2019 at 13:22, Upendra Yadav <up...@gmail.com> wrote:
>>
>>> Hi,
>>> I have initialised kafka consumer:
>>> KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<byte[],
>>> byte[]>(consumerConfig);
>>>
>>> and subscribed with topic pattern:
>>> consumer.subscribe(Pattern.compile(topicRegex), listener);
>>>
>>> now, I'm trying to get the list of topics.
>>> Map<String, List<PartitionInfo>> topicsPartitions =
>> consumer.listTopics();
>>>
>>> here topic list is not following given pattern. it is giving all topic
>>> list.
>>> Is this expected behaviour? Is there any API to get subscribed topic list
>>> with consumer instance?
>>> or Do I need to write simple prog to process this list with same pattern?
>>>
>>
>
Re: list of pattern processed topic list
Posted by Kamal Chandraprakash <ka...@gmail.com>.
You can use the KafkaConsumer#assignment() method to get all the assigned
topic-partitions for that consumer instance.
But, you've to periodically call poll method to get the latest assignment
which may return records.
This shortcoming is actively discussed in the below threads.
https://mail-archives.apache.org/mod_mbox/kafka-dev/201908.mbox/<08030a68-3f0b-42db-9b79-dfcd3200cf25%40www.fastmail.com>
<https://mail-archives.apache.org/mod_mbox/kafka-dev/201908.mbox/%3C08030a68-3f0b-42db-9b79-dfcd3200cf25%40www.fastmail.com%3E>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-505%3A+Add+new+public+method+to+only+update+assignment+metadata+in+consumer
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97551484
On Mon, Aug 19, 2019 at 6:33 PM M. Manna <ma...@gmail.com> wrote:
> Kafka doesn’t have regex subscription. You could get a list of available
> topic using admin client. From there, you can decide what topics you would
> like to subscribe to (or process using your logic).
>
> Check documentation for AdminClient.listTopics and ListTopicResult holder.
>
> I hope this helps.
>
> Regards,
>
> On Mon, 19 Aug 2019 at 13:22, Upendra Yadav <up...@gmail.com> wrote:
>
> > Hi,
> > I have initialised kafka consumer:
> > KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<byte[],
> > byte[]>(consumerConfig);
> >
> > and subscribed with topic pattern:
> > consumer.subscribe(Pattern.compile(topicRegex), listener);
> >
> > now, I'm trying to get the list of topics.
> > Map<String, List<PartitionInfo>> topicsPartitions =
> consumer.listTopics();
> >
> > here topic list is not following given pattern. it is giving all topic
> > list.
> > Is this expected behaviour? Is there any API to get subscribed topic list
> > with consumer instance?
> > or Do I need to write simple prog to process this list with same pattern?
> >
>
Re: list of pattern processed topic list
Posted by "M. Manna" <ma...@gmail.com>.
Kafka doesn’t have regex subscription. You could get a list of available
topic using admin client. From there, you can decide what topics you would
like to subscribe to (or process using your logic).
Check documentation for AdminClient.listTopics and ListTopicResult holder.
I hope this helps.
Regards,
On Mon, 19 Aug 2019 at 13:22, Upendra Yadav <up...@gmail.com> wrote:
> Hi,
> I have initialised kafka consumer:
> KafkaConsumer<byte[], byte[]> consumer = new KafkaConsumer<byte[],
> byte[]>(consumerConfig);
>
> and subscribed with topic pattern:
> consumer.subscribe(Pattern.compile(topicRegex), listener);
>
> now, I'm trying to get the list of topics.
> Map<String, List<PartitionInfo>> topicsPartitions = consumer.listTopics();
>
> here topic list is not following given pattern. it is giving all topic
> list.
> Is this expected behaviour? Is there any API to get subscribed topic list
> with consumer instance?
> or Do I need to write simple prog to process this list with same pattern?
>