[jira] [Created] (KAFKA-14455) Kafka Connect create and update REST APIs should surface failures while writing to the config topic

["Yash Mayya (Jira)" <ji...@apache.org>]

Yash Mayya created KAFKA-14455:
----------------------------------

             Summary: Kafka Connect create and update REST APIs should surface failures while writing to the config topic
                 Key: KAFKA-14455
                 URL: https://issues.apache.org/jira/browse/KAFKA-14455
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
            Reporter: Yash Mayya
            Assignee: Yash Mayya


Kafka Connect's `POST /connectors` and `PUT /connectors/\{connector}/config` REST APIs internally simply write a message to the Connect cluster's internal config topic (which is then processed asynchronously by the herder). However, no callback is passed to the producer's send method and there is no error handling in place for producer send failures (see [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716] / [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726]).

Consider one such case where the Connect worker's principal doesn't have a WRITE ACL on the cluster's config topic. Now suppose the user submits a connector's configs via one of the above two APIs. The producer send [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L716] / [here|https://github.com/apache/kafka/blob/c1a54671e8fc6c7daec5f5ec3d8c934be96b4989/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L726] won't succeed (due to a TopicAuthorizationException) but the API responses will be `201 Created` success responses anyway. This is a very poor UX because the connector will actually never be created but the API response indicated success. Furthermore, this failure would only be detectable if TRACE logs are enabled (via [this log)|https://github.com/apache/kafka/blob/df29b17fc40f7c15460988d58bc652c3d66b60f8/clients/src/main/java/org/apache/kafka/clients/producer/internals/ProducerBatch.java] making it near impossible for users to debug. Producer callbacks should be used to surface write failures back to the user via the API response.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)