Kafka consumer questoin.

[Shrikant Patel <SP...@pdxinc.com>]

Hi

As I understand the producer in Kafka connect to broker list first to fetch the metadata. Procedure uses that data to directly connect to leader for partition that it's trying to publish to.

From my understanding of Kafka protocol and following other threads, the consumer so will be doing the same as producer. Meaning after the partition is assigned to a consumer, the consumer will directly connect to leader to read the message.

Could provide some insight to that communication process (after partition is assigned to consumer) between the broker and consumer. And if partition leadership changes, how consumers become aware of that.

Thanks in advance
Shri

This e-mail and its contents (to include attachments) are the property of National Health Systems, Inc., its subsidiaries and affiliates, including but not limited to Rx.com Community Healthcare Network, Inc. and its subsidiaries, and may contain confidential and proprietary or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that any unauthorized disclosure, copying, or distribution of this e-mail or of its attachments, or the taking of any unauthorized action based on information contained herein is strictly prohibited. Unauthorized use of information contained herein may subject you to civil and criminal prosecution and penalties. If you are not the intended recipient, please immediately notify the sender by telephone at 800-433-5719 or return e-mail and permanently delete the original e-mail.

Re: Kafka consumer questoin.

[Mickael Maison <mi...@gmail.com>]

Hi Shri,

I suggest you have a look at the protocol documentation and more
specifically at this section:
http://kafka.apache.org/protocol.html#protocol_partitioning

Regarding your question:
The consumer does the same logic as the producer to get started. It
sends a Metadata request to the bootstrap servers you provided as
configuration. From there, it's able to find which brokers are the
leaders for the partitions it wants to subscribe to. Then to get
messages, it sends a Fetch request to the corresponding leader. If
leadership changes, the previous leader will return an error
indicating it is not the leader anymore (NOT_LEADER_FOR_PARTITION). To
find the new leader, the Consumer will send a new Metadata request.


On Thu, Aug 31, 2017 at 1:15 PM, Shrikant Patel <SP...@pdxinc.com> wrote:
> Hi
>
> As I understand the producer in Kafka connect to broker list first to fetch the metadata. Procedure uses that data to directly connect to leader for partition that it's trying to publish to.
>
> From my understanding of Kafka protocol and following other threads, the consumer so will be doing the same as producer. Meaning after the partition is assigned to a consumer, the consumer will directly connect to leader to read the message.
>
> Could provide some insight to that communication process (after partition is assigned to consumer) between the broker and consumer. And if partition leadership changes, how consumers become aware of that.
>
> Thanks in advance
> Shri
>
> This e-mail and its contents (to include attachments) are the property of National Health Systems, Inc., its subsidiaries and affiliates, including but not limited to Rx.com Community Healthcare Network, Inc. and its subsidiaries, and may contain confidential and proprietary or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that any unauthorized disclosure, copying, or distribution of this e-mail or of its attachments, or the taking of any unauthorized action based on information contained herein is strictly prohibited. Unauthorized use of information contained herein may subject you to civil and criminal prosecution and penalties. If you are not the intended recipient, please immediately notify the sender by telephone at 800-433-5719 or return e-mail and permanently delete the original e-mail.