ConsumerConnector and 0.10 message format

[Jeff Pollard <je...@gmail.com>]

Hello all,

We are in the process of upgrading all our consumers to version 0.10+. One
of our pre-0.10 consumers still uses the deprecated ConsumerConnector API
(I believe colloquially called the "old consumer").

We're unfortunately not able to upgrade this consumer to the new consumer
API, as the consumer is baked into a larger piece of software we did not
write, and they have not offered a version of it with the new consumer.
They have, however, released a version using Kafka 0.10, which we have
upgraded to.

We also have upgraded our *brokers* to 0.10+, but per the potential
performance impact
<http://kafka.apache.org/0102/documentation.html#upgrade_10_performance_impact>
noted
in documentation have set the log.message.format.version to 0.8.2. This has
worked well for us, as we've slowly been upgrading our consumers to 0.10+.

My question is about using the old ConsumerConnector API with Kafka 0.10+
client and a 0.10+ broker. From my reading of the documentation, it seems
the performance impact is translating the message format by the broker, and
is independent of the kind (old or new) consumer. I believe both the old
and new consumer both use the same fetch API, but I hadn't verified that
yet.

We just wanted to make sure this was correct, as to avoid any unexpected
performance impacts when we finish our upgrade and change the on-broker
message format to the 0.10+ one.

Thanks again. Happy to answer or clarify any points of this email as needed.

Re: ConsumerConnector and 0.10 message format

[Ismael Juma <is...@juma.me.uk>]

The performance impact happens if the consumer doesn't support the message
format defined by log.message.format.version. The old consumer in Kafka
0.10 supports the message format introduced in 0.10 (including timestamps
and the ability to avoid recompression in the produce path) so what you
propose is fine.

However, the old consumer does _not_ support the message format introduced
in 0.11.0 (no matter the version) so you will eventually have to deal with
down conversion if that application cannot be upgraded to use the Java
consumer. Kafka 2.0.0 will contain down conversion efficiency improvements
and you may consider testing if the down conversion costs are bearable for
that application. If they are, the 0.11.0+ message format version is a good
path forward as it is necessary for using headers, transactions and the
idempotent producer.

Ismael

On Thu, Jun 21, 2018 at 11:30 PM Jeff Pollard <je...@gmail.com>
wrote:

> Hello all,
>
> We are in the process of upgrading all our consumers to version 0.10+. One
> of our pre-0.10 consumers still uses the deprecated ConsumerConnector API
> (I believe colloquially called the "old consumer").
>
> We're unfortunately not able to upgrade this consumer to the new consumer
> API, as the consumer is baked into a larger piece of software we did not
> write, and they have not offered a version of it with the new consumer.
> They have, however, released a version using Kafka 0.10, which we have
> upgraded to.
>
> We also have upgraded our *brokers* to 0.10+, but per the potential
> performance impact
> <
> http://kafka.apache.org/0102/documentation.html#upgrade_10_performance_impact
> >
> noted
> in documentation have set the log.message.format.version to 0.8.2. This has
> worked well for us, as we've slowly been upgrading our consumers to 0.10+.
>
> My question is about using the old ConsumerConnector API with Kafka 0.10+
> client and a 0.10+ broker. From my reading of the documentation, it seems
> the performance impact is translating the message format by the broker, and
> is independent of the kind (old or new) consumer. I believe both the old
> and new consumer both use the same fetch API, but I hadn't verified that
> yet.
>
> We just wanted to make sure this was correct, as to avoid any unexpected
> performance impacts when we finish our upgrade and change the on-broker
> message format to the 0.10+ one.
>
> Thanks again. Happy to answer or clarify any points of this email as
> needed.
>