[jira] [Created] (KAFKA-14873) Pluggable storage for Kafka Connect internal topics

["Malthe Borch (Jira)" <ji...@apache.org>]

Malthe Borch created KAFKA-14873:
------------------------------------

             Summary: Pluggable storage for Kafka Connect internal topics
                 Key: KAFKA-14873
                 URL: https://issues.apache.org/jira/browse/KAFKA-14873
             Project: Kafka
          Issue Type: Improvement
            Reporter: Malthe Borch


The Kafka Connect framework relies on compacted topics to store config, offset and status information for each connector.

This conflates two kinds of data, control and content, which some people disagree with. Notably, [Azure Event Hub|https://learn.microsoft.com/en-us/azure/event-hubs/log-compaction] does not (or _did not_, because there's currently a preview release out which does have support for compacted topics albeit only at the more expensive premium tiers).

In some deployments, it may be desirable to use a different backend for these control settings (which essentially take a key/value form), for example [Azure Table Storage|https://learn.microsoft.com/en-us/rest/api/storageservices/table-service-rest-api] – basically any key/value store that provides the Write-If-Matches primitive to update a key only if the current value matches a known value.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)