You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Aminouvic <am...@gmail.com> on 2019/09/13 10:23:29 UTC

__consumer_offsets compaction settings / metadata performance

Hi all, 

Currently with default settings min.cleanable.dirty.ratio/log.cleaner.min.cleanable.ratio (0.5) __consumer_offsets partition size grows up to 80Mb and this seems to slow down metadata related operations (notably commits / join group).
What are the normal/expected partition avg size for the __consumer_offsets topic ?
Are there any recommendations to settings min.cleanable.dirty.ratio/log.cleaner.min.cleanable.ratio for the __consumer_offsets topic ?

Kafka Cluster 
	• version 0.10.1
	• 6 nodes 
	• __consumer_offsets #partitions : 50 
	• Cleaner config :
		○ min.cleanable.dirty.ratio : 0.5
		○ log.cleaner.enable : True
		○ log.cleaner.io.buffer.size : 524288
		○ log.cleaner.dedupe.buffer.size : 1073741824
		○ log.cleaner.io.max.bytes.per.second: 104857600

Regards,
Amine