Potential message incompatibility between versions 0.11 and 2.2.1 of Kafka?

[Malcolm McFarland <mm...@cavulus.com>]

Hey folks,

We're using Samza 0.14.1 for a project, with Kafka 0.11.0.2 as a client.
We're also using Amazon's MSK for our clusters, which is currently set to
Kafka broker version 2.2.1. Occasionally, when starting Samza containers,
we'll see the beginnings of an attempt to read Samza's checkpoint stream,
but this will halt after the first message with no errors:

2020-04-09T19:24:14.968Z Validating offset 0 for topic and partition
[__samza_checkpoint_ver_1_for_<job_name>_1,0]
2020-04-09T19:24:14.969Z Able to successfully read from offset 0 for topic
and partition [__samza_checkpoint_ver_1_for_<job_name>_1,0]. Using it to
instantiate consumer.
2020-04-09T19:24:14.974Z Reading checkpoint for taskName
SystemStreamPartition [kafka, <topic>, <partition>]

During debugging, I accidentally used 2 different versions of kafkacat, and
noticed that librdkafka 1.3.0 *can* read the checkpoint stream, but
librdkafka 0.8.6 *cannot*. Could there be a version incompatibility with
how the data is being stored on the kafka server?

Cheers,
Malcolm McFarland
Cavulus