Connect/broker compatibility

[Chris Egerton <fe...@gmail.com>]

Hi all,

I noticed that we recently altered Kafka Connect to explicitly disable
idempotent producers by default [1], with the rationale being that this
ensured compatibility with brokers running versions older than 0.11 and
ACL-protected brokers running versions older than 2.8. There has also been
mention that KIP-318 [2] would have to be approved before reverting this
change and enabling idempotent producers by default on Kafka Connect.

This is a pretty restrictive precedent to set and I want to see if we could
consider relaxing it a little bit. When KIP-679 [3] was approved and
idempotence was enabled by default in all producers starting with 3.0, it
was with full understanding that it would lead to breaking changes in these
scenarios for all producer applications, and with conviction that the
upgrade steps laid out were sufficient for users to be able to handle this
gracefully. Kafka Connect is one such producer application, and it already
gives users the means to implement the upgrade steps laid out in KIP-679 in
order to remain compatible with older broker versions.

Given that it's possible for users to reconfigure their clusters to work
properly with older broker versions, the changes in KIP-679 were released
in a major version bump (3.0.0) that permits breaking of backwards
compatibility, and there is no significant operational burden placed on
users of Kafka Connect by KIP-679 that isn't also placed on other producer
applications, do we really need to go through the process of yet another
KIP in order to apply these changes to Kafka Connect? And if so, what
differentiates Kafka Connect from all the other producer applications that
were affected by KIP-679, including Kafka Streams applications configured
with the (default) processing mode of "at least once"?

Instead of choosing to emphasize cross-major-version backwards
compatibility between Kafka Connect and Kafka brokers, we might instead
invest our efforts in creating more detailed upgrade notes that call out
changes like these and let users know exactly how to configure their Kafka
Connect workers (and producer applications in general) when upgrading Kafka
Connect and/or their Kafka brokers.

[1] - https://issues.apache.org/jira/browse/KAFKA-13759
[2] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent
[3] -
https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default

Cheers,

Chris