Kafka - producer writing to a certain broker...

[Rajib Deb <Ra...@infosys.com>]

Hi,
I came across the below question and wanted to seek an answer on the same.

If a producer needs to write to a certain broker only, is this possible. For example, if the producer is in Europe, it will write to the broker near to Europe, if US it will write to broker near to US. But consumers should be able to read from both the topics.

Thanks
Rajib

Re: Kafka - producer writing to a certain broker...

[Ricardo Ferreira <ri...@riferrei.com>]

I am not a huge fan of criticizing without making myself useful first; 
so here is what you can do in order to have a producer writing records 
to a specific broker:

1. Create a topic with the # of partitions equal to the # of brokers. By 
default Kafka will try to evenly distributed the partitions across the 
brokers so it is expected that each broker will host at least one 
partition of the topic.

2. Figure out which broker host the leader of the partition. Each 
partition will always have one leader and multiple replicas. The leader 
(where all the writes goes by) will live in one of the brokers. You can 
figure this out by running the `kafka-topics` command using the option 
`--describe`. Your job here is figuring the partition id and not the 
broker id.

3. In the producer code, send records using one of the following approach:

     ProducerRecord<K, V> record = new ProducerRecord(topic, 
*partitionId*, K, V);

     producer.send(record);

4. There might be situations in which the leader of a given partition 
may be moved to another broker, in which you will have to keep track of 
this movement and change your code accordingly. You can reasonably 
achieve a dynamic behavior in your producer code by fetching the 
partitions using Kafka's admin API before executing the send() command 
to check if things have changed.

As you probably learn at this point, in Kafka the unit of writing is not 
the broker but a partition that in turn lives in one of the brokers.

Not here comes the critic:

- *Don't Do This!*

You will be giving up of all the logic that Kafka puts in place to 
ensure horizontal scalability and fault-tolerance. This practice might 
be common in some messaging technologies but it isn't in Kafka. Not to 
mention that you would have to have an considerable Ops effort to keep 
track of partitions in brokers, as well as write a more sophisticated 
code to protect you against situations that would easily break all this 
logic such as growing/shrinking the # of brokers and/or partitions. 
Likely, Kafka is not the appropriate technology for you in this case.

Thanks,

-- Ricardo

On 7/26/20 11:11 AM, Rajib Deb wrote:
> Hi,
> I came across the below question and wanted to seek an answer on the same.
>
> If a producer needs to write to a certain broker only, is this possible. For example, if the producer is in Europe, it will write to the broker near to Europe, if US it will write to broker near to US. But consumers should be able to read from both the topics.
>
> Thanks
> Rajib

Kafka - producer writing to a certain broker...

[Rajib Deb <Ra...@infosys.com>]

Hi,
I came across the below question and wanted to seek an answer on the same.

If a producer needs to write to a certain broker only, is this possible. For example, if the producer is in Europe, it will write to the broker near to Europe, if US it will write to broker near to US. But consumers should be able to read from both the topics.

Thanks
Rajib

Re: Kafka - producer writing to a certain broker...

[Ma...@cognizant.com]

You should use active-active mirror maker

On 7/25/20, 9:03 AM, "Rajib Deb" <Ra...@infosys.com> wrote:

    [External]


    Hi,
    I came across the below question and wanted to seek an answer on the same.

    If a producer needs to write to a certain broker only, is this possible. For example, if the producer is in Europe, it will write to the broker near to Europe, if US it will write to broker near to US. But consumers should be able to read from both the topics.

    Thanks
    Rajib


This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.
This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful. Where permitted by applicable law, this e-mail and other e-mail communications sent to and from Cognizant e-mail addresses may be monitored.