Whether kafka broker will have impact with 2 MB message size

[Hemnath K B <kb...@gmail.com>]

We have a use case where we want to produce data to kafka with max
size of 2 MB rarely (That is, based on user operations message size
will vary).

Whether producing 2 Mb size will have any impact or we need to split
the message to small chunk such as 100 KB and produce.

If we produce into small chunk, it will increase response time for the
user. Also, we have checked by producing 2 MB message to kafka and we
doesn't see much latency there.

Anyway if we split the data and produce, it doesn't have any impact in
disk size. But whether broker performance will degrade due to this?

Our broker configuration is:

RAM 125.6 GB Disk Size 2.9 TB Processors 40

Thanks,
Hemnath K B.

Re: Whether kafka broker will have impact with 2 MB message size

["1095193290@qq.com" <10...@qq.com>]

We have a use case where we want to produce data to kafka with max
size of 2 MB rarely (That is, based on user operations message size
will vary).

Whether producing 2 Mb size will have any impact or we need to split
the message to small chunk such as 100 KB and produce.

If we produce into small chunk, it will increase response time for the
user. Also, we have checked by producing 2 MB message to kafka and we
doesn't see much latency there.

Anyway if we split the data and produce, it doesn't have any impact in
disk size. But whether broker performance will degrade due to this?

Our broker configuration is:

RAM 125.6 GB Disk Size 2.9 TB Processors 40

Thanks,
Hemnath K B.

Re: Whether kafka broker will have impact with 2 MB message size

[Peter Bukowinski <pm...@gmail.com>]

We have many production clusters with three topics in the 1-3MB range and the rest in the multi-kb to sub-kb range. We do use gzip compression, implemented at the broker rather than the producer level. The clusters don’t usually break a sweat. We use MirrorMaker to aggregate these topics to a large central cluster and the message sizes aren’t an issue there, either.

The main problem we have with these large topics — especially one of them that is high throughput — is that after a cluster has been up a while and partitions have moved around (we use cruise-control) due to hardware failures, the topic tends to become less balanced across the broker storage and some brokers/disks fill up faster than others.

—
Peter

> On Mar 13, 2019, at 6:28 PM, 1095193290@qq.com wrote:
> 
> We have a use case where we want to produce data to kafka with max
> size of 2 MB rarely (That is, based on user operations message size
> will vary).
> 
> Whether producing 2 Mb size will have any impact or we need to split
> the message to small chunk such as 100 KB and produce.
> 
> If we produce into small chunk, it will increase response time for the
> user. Also, we have checked by producing 2 MB message to kafka and we
> doesn't see much latency there.
> 
> Anyway if we split the data and produce, it doesn't have any impact in
> disk size. But whether broker performance will degrade due to this?
> 
> Our broker configuration is:
> 
> RAM 125.6 GB Disk Size 2.9 TB Processors 40
> 
> Thanks,
> Hemnath K B.