Produce Message Conversions Per Sec

[Peter Bukowinski <pm...@gmail.com>]

Greetings,

I have a concern about the produce message conversions per sec metrics of my kafka brokers. I have a go application that produces topics into a kafka 2.0.1 cluster using confluent-kafka-go with librdkafka 0.11.6 (and log.message.format.version=2.0). The produce conversions/sec metrics are identical to the messages in/sec metric, which indicates all messages produced into this cluster are undergoing some sort of format conversion. I recently upgraded my brokers from kafka 1.0.1 and my clients from the 0.11.4 version of the library, but that had no effect on the produce conversion rate.

One thing to note in my configuration is that the producers are not configured to compress the messages, but compression.type is set to gzip on the brokers. In my scenario, I’d rather my brokers spend the CPU cycles on compression than my client. Is this responsible for the conversion metrics I’m seeing? If so, I’ll stop worrying since it is working as intended.

—
Peter Bukowinski