You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by SenthilKumar K <se...@gmail.com> on 2020/05/05 18:43:53 UTC
KafkaConsumer.partitionsFor() Vs KafkaAdminClient.describeTopics()
Hi Team, We are using KafkaConsumer.partitionsFor() API to find the list of
available partitions. After fetching the list of partitions, We use
Consumer.offsetsForTimes() API to find the offsets for a given timestamp.
The API Consumer.partitionsFor() simply returning all partitions including
the partitions which the leader is set to -1. It's causing an issue
(Timeout Exception) when we call Consumer.offsetsForTimes() API.
I'm planning to use adminClient.describeTopics(list).all().get(); And
filter only the partitions which are healthy. Will there be any performance
impact of using AdminClient?
Kafka Version: 2.4.1
Kafka Client: 2.3.0
--Senthil
Re: KafkaConsumer.partitionsFor() Vs KafkaAdminClient.describeTopics()
Posted by SenthilKumar K <se...@gmail.com>.
Thanks Manoj. It makes sense to use Consumer itself for fetching meta data.
On Wed, May 6, 2020 at 12:39 AM <Ma...@cognizant.com> wrote:
>
>
>
>
> Glade , it work for you .
>
>
>
> Kafka Admin API run on zookeeper and sometime you don’t have access to
> Zookeeper host /port . I don’t know in your scenario how you are managing
> kafka/Zk cluster but for security purpose , Zookeeper access only limited
> to kafka Cluster .
>
>
>
>
>
> *From: *SenthilKumar K <se...@gmail.com>
> *Date: *Tuesday, May 5, 2020 at 12:06 PM
> *To: *"Agrawal, Manoj (Cognizant)" <Ma...@cognizant.com>,
> Senthil kumar <se...@gmail.com>
> *Cc: *"users@kafka.apache.org" <us...@kafka.apache.org>, "
> senthilec566@apache.org" <se...@apache.org>
> *Subject: *Re: KafkaConsumer.partitionsFor() Vs
> KafkaAdminClient.describeTopics()
>
>
>
> *[External]*
>
> Thanks Manoj. It works for me.
>
>
>
> Looks to me the KafkaAdminClient (Singleton instance ) is faster than
> Consumer.partitionsFor() API. In terms of performance which one is good to
> fetch the metadata of a given topic. Thanks!
>
>
>
> On Wed, May 6, 2020 at 12:26 AM <Ma...@cognizant.com> wrote:
>
> I think you can filter list of topic return by
> KafkaConsumer.partitionsFor() on by using method type , if this is
> PartitionInfo.leader() then include those partition in list .
>
>
>
> On 5/5/20, 11:44 AM, "SenthilKumar K" <se...@gmail.com> wrote:
>
> [External]
>
>
> Hi Team, We are using KafkaConsumer.partitionsFor() API to find the
> list of
> available partitions. After fetching the list of partitions, We use
> Consumer.offsetsForTimes() API to find the offsets for a given
> timestamp.
>
> The API Consumer.partitionsFor() simply returning all partitions
> including
> the partitions which the leader is set to -1. It's causing an issue
> (Timeout Exception) when we call Consumer.offsetsForTimes() API.
>
> I'm planning to use adminClient.describeTopics(list).all().get(); And
> filter only the partitions which are healthy. Will there be any
> performance
> impact of using AdminClient?
>
> Kafka Version: 2.4.1
> Kafka Client: 2.3.0
>
> --Senthil
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored. This e-mail and any files
> transmitted with it are for the sole use of the intended recipient(s) and
> may contain confidential and privileged information. If you are not the
> intended recipient(s), please reply to the sender and destroy all copies of
> the original message. Any unauthorized review, use, disclosure,
> dissemination, forwarding, printing or copying of this email, and/or any
> action taken in reliance on the contents of this e-mail is strictly
> prohibited and may be unlawful. Where permitted by applicable law, this
> e-mail and other e-mail communications sent to and from Cognizant e-mail
> addresses may be monitored.
>
Re: KafkaConsumer.partitionsFor() Vs
KafkaAdminClient.describeTopics()
Posted by Ma...@cognizant.com.
Glade , it work for you .
Kafka Admin API run on zookeeper and sometime you don’t have access to Zookeeper host /port . I don’t know in your scenario how you are managing kafka/Zk cluster but for security purpose , Zookeeper access only limited to kafka Cluster .
From: SenthilKumar K <se...@gmail.com>
Date: Tuesday, May 5, 2020 at 12:06 PM
To: "Agrawal, Manoj (Cognizant)" <Ma...@cognizant.com>, Senthil kumar <se...@gmail.com>
Cc: "users@kafka.apache.org" <us...@kafka.apache.org>, "senthilec566@apache.org" <se...@apache.org>
Subject: Re: KafkaConsumer.partitionsFor() Vs KafkaAdminClient.describeTopics()
[External]
Thanks Manoj. It works for me.
Looks to me the KafkaAdminClient (Singleton instance ) is faster than Consumer.partitionsFor() API. In terms of performance which one is good to fetch the metadata of a given topic. Thanks!
On Wed, May 6, 2020 at 12:26 AM <Ma...@cognizant.com>> wrote:
I think you can filter list of topic return by KafkaConsumer.partitionsFor() on by using method type , if this is PartitionInfo.leader() then include those partition in list .
On 5/5/20, 11:44 AM, "SenthilKumar K" <se...@gmail.com>> wrote:
[External]
Hi Team, We are using KafkaConsumer.partitionsFor() API to find the list of
available partitions. After fetching the list of partitions, We use
Consumer.offsetsForTimes() API to find the offsets for a given timestamp.
The API Consumer.partitionsFor() simply returning all partitions including
the partitions which the leader is set to -1. It's causing an issue
(Timeout Exception) when we call Consumer.offsetsForTimes() API.
I'm planning to use adminClient.describeTopics(list).all().get(); And
filter only the partitions which are healthy. Will there be any performance
impact of using AdminClient?
Kafka Version: 2.4.1
Kafka Client: 2.3.0
--Senthil
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored. This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
Re: KafkaConsumer.partitionsFor() Vs KafkaAdminClient.describeTopics()
Posted by SenthilKumar K <se...@gmail.com>.
Thanks Manoj. It works for me.
Looks to me the KafkaAdminClient (Singleton instance ) is faster than
Consumer.partitionsFor() API. In terms of performance which one is good to
fetch the metadata of a given topic. Thanks!
On Wed, May 6, 2020 at 12:26 AM <Ma...@cognizant.com> wrote:
> I think you can filter list of topic return by
> KafkaConsumer.partitionsFor() on by using method type , if this is
> PartitionInfo.leader() then include those partition in list .
>
>
>
> On 5/5/20, 11:44 AM, "SenthilKumar K" <se...@gmail.com> wrote:
>
> [External]
>
>
> Hi Team, We are using KafkaConsumer.partitionsFor() API to find the
> list of
> available partitions. After fetching the list of partitions, We use
> Consumer.offsetsForTimes() API to find the offsets for a given
> timestamp.
>
> The API Consumer.partitionsFor() simply returning all partitions
> including
> the partitions which the leader is set to -1. It's causing an issue
> (Timeout Exception) when we call Consumer.offsetsForTimes() API.
>
> I'm planning to use adminClient.describeTopics(list).all().get(); And
> filter only the partitions which are healthy. Will there be any
> performance
> impact of using AdminClient?
>
> Kafka Version: 2.4.1
> Kafka Client: 2.3.0
>
> --Senthil
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful. Where permitted by
> applicable law, this e-mail and other e-mail communications sent to and
> from Cognizant e-mail addresses may be monitored.
>
Re: KafkaConsumer.partitionsFor() Vs
KafkaAdminClient.describeTopics()
Posted by Ma...@cognizant.com.
I think you can filter list of topic return by KafkaConsumer.partitionsFor() on by using method type , if this is PartitionInfo.leader() then include those partition in list .
On 5/5/20, 11:44 AM, "SenthilKumar K" <se...@gmail.com> wrote:
[External]
Hi Team, We are using KafkaConsumer.partitionsFor() API to find the list of
available partitions. After fetching the list of partitions, We use
Consumer.offsetsForTimes() API to find the offsets for a given timestamp.
The API Consumer.partitionsFor() simply returning all partitions including
the partitions which the leader is set to -1. It's causing an issue
(Timeout Exception) when we call Consumer.offsetsForTimes() API.
I'm planning to use adminClient.describeTopics(list).all().get(); And
filter only the partitions which are healthy. Will there be any performance
impact of using AdminClient?
Kafka Version: 2.4.1
Kafka Client: 2.3.0
--Senthil
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.