You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by yong <sh...@gmail.com> on 2018/11/05 04:24:17 UTC

Kafka ingestion data not equally distribute among brokers

Hi Kafka experts,

We have a 3 nodes kafka setup with data streams from mysql binlog, we
noticed that from grafana monitoring in and out, some broker nodes are
having lower data ingested in and out, is there anyway to balance it? Or it
is nature of Kafka?

Thanks!

Re: Kafka ingestion data not equally distribute among brokers

Posted by Manoj Khangaonkar <kh...@gmail.com>.

Hi

That is part of the Producer API.

For Java see
https://kafka.apache.org/20/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
ProducerRecord<K,V>

regards



On Tue, Nov 6, 2018 at 5:46 PM Shiuh Rong Yong <
shiuhrongyong@cirosolution.com> wrote:

> Hi Manoj,
>
> I am new on Kafka, is it something mentioned on this site?
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/streams/state/KeyValueStore.html
>
> Where can we set the key and value in Kafka?
>
> Thanks!
>
>
> ----- Original Message -----
> From: "Manoj Khangaonkar" <kh...@gmail.com>
> To: "users" <us...@kafka.apache.org>
> Sent: Tuesday, November 6, 2018 4:36:40 AM
> Subject: Re: Kafka ingestion data not equally distribute among brokers
>
> Hi
>
> In Kafka topic data is split into partitions. Partitions are assigned to
> brokers.
>
> I assume what you are trying to says is that the distribution of messages
> across partitions is not balanced.
>
> Messages are written to topics as key,value. The messages are distributed
> across partitions based on key.
>
> Perhaps you need to pick a better key that gives balanced distribution.
>
> regards
>
>
>
> On Mon, Nov 5, 2018 at 9:22 AM yong <sh...@gmail.com> wrote:
>
> > Hi Kafka experts,
> >
> > We have a 3 nodes kafka setup with data streams from mysql binlog, we
> > noticed that from grafana monitoring in and out, some broker nodes are
> > having lower data ingested in and out, is there anyway to balance it? Or
> it
> > is nature of Kafka?
> >
> > Thanks!
> >
>
>
> --
> http://khangaonkar.blogspot.com/
>


-- 
http://khangaonkar.blogspot.com/

Re: Kafka ingestion data not equally distribute among brokers

Posted by Shiuh Rong Yong <sh...@cirosolution.com>.

Hi Manoj,

I am new on Kafka, is it something mentioned on this site? https://kafka.apache.org/0100/javadoc/org/apache/kafka/streams/state/KeyValueStore.html

Where can we set the key and value in Kafka?

Thanks!

----- Original Message -----
From: "Manoj Khangaonkar" <kh...@gmail.com>
To: "users" <us...@kafka.apache.org>
Sent: Tuesday, November 6, 2018 4:36:40 AM
Subject: Re: Kafka ingestion data not equally distribute among brokers

Hi

In Kafka topic data is split into partitions. Partitions are assigned to
brokers.

I assume what you are trying to says is that the distribution of messages
across partitions is not balanced.

Messages are written to topics as key,value. The messages are distributed
across partitions based on key.

Perhaps you need to pick a better key that gives balanced distribution.

regards

On Mon, Nov 5, 2018 at 9:22 AM yong <sh...@gmail.com> wrote:

> Hi Kafka experts,
>
> We have a 3 nodes kafka setup with data streams from mysql binlog, we
> noticed that from grafana monitoring in and out, some broker nodes are
> having lower data ingested in and out, is there anyway to balance it? Or it
> is nature of Kafka?
>
> Thanks!
>

-- 
http://khangaonkar.blogspot.com/

Re: Kafka ingestion data not equally distribute among brokers

Posted by Manoj Khangaonkar <kh...@gmail.com>.

Hi

In Kafka topic data is split into partitions. Partitions are assigned to
brokers.

I assume what you are trying to says is that the distribution of messages
across partitions is not balanced.

Messages are written to topics as key,value. The messages are distributed
across partitions based on key.

Perhaps you need to pick a better key that gives balanced distribution.

regards

On Mon, Nov 5, 2018 at 9:22 AM yong <sh...@gmail.com> wrote:

> Hi Kafka experts,
>
> We have a 3 nodes kafka setup with data streams from mysql binlog, we
> noticed that from grafana monitoring in and out, some broker nodes are
> having lower data ingested in and out, is there anyway to balance it? Or it
> is nature of Kafka?
>
> Thanks!
>

-- 
http://khangaonkar.blogspot.com/