Kafka log.message.format.version and consumer client versions

[Dave Hamilton <dh...@nanigans.com>]

Hi, I have a question about the performance implications of upgrading the Kafka message format relating to the following from the upgrade documentation:

The message format in 0.10.0 includes a new timestamp field and uses relative offsets for compressed messages. The on disk message format can be configured through log.message.format.version in the server.properties file. The default on-disk message format is 0.10.0. If a consumer client is on a version before 0.10.0.0, it only understands message formats before 0.10.0. In this case, the broker is able to convert messages from the 0.10.0 format to an earlier format before sending the response to the consumer on an older version. However, the broker can't use zero-copy transfer in this case.

Would it be sufficient to upgrade all consumers to the new client version (both the Java and Scala clients) to ensure zero-copy transfer is still used after upgrading the message version? Or do all consumers using the Scala API need to be switched to using the new Java consumer API?

Thanks,
Dave

Re: Kafka log.message.format.version and consumer client versions

[Manikumar <ma...@gmail.com>]

yes, It is sufficient to upgrade all consumers to new version. No need to
switch Scala APIs to Java APIs.

On Mon, May 8, 2017 at 10:03 PM, Dave Hamilton <dh...@nanigans.com>
wrote:

> Hi, I have a question about the performance implications of upgrading the
> Kafka message format relating to the following from the upgrade
> documentation:
>
> The message format in 0.10.0 includes a new timestamp field and uses
> relative offsets for compressed messages. The on disk message format can be
> configured through log.message.format.version in the server.properties
> file. The default on-disk message format is 0.10.0. If a consumer client is
> on a version before 0.10.0.0, it only understands message formats before
> 0.10.0. In this case, the broker is able to convert messages from the
> 0.10.0 format to an earlier format before sending the response to the
> consumer on an older version. However, the broker can't use zero-copy
> transfer in this case.
>
> Would it be sufficient to upgrade all consumers to the new client version
> (both the Java and Scala clients) to ensure zero-copy transfer is still
> used after upgrading the message version? Or do all consumers using the
> Scala API need to be switched to using the new Java consumer API?
>
> Thanks,
> Dave
>