[jira] [Created] (KAFKA-15470) Allow creating connectors in a stopped state

["Yash Mayya (Jira)" <ji...@apache.org>]

Yash Mayya created KAFKA-15470:
----------------------------------

             Summary: Allow creating connectors in a stopped state
                 Key: KAFKA-15470
                 URL: https://issues.apache.org/jira/browse/KAFKA-15470
             Project: Kafka
          Issue Type: New Feature
          Components: connect, KafkaConnect
            Reporter: Yash Mayya
            Assignee: Yash Mayya
             Fix For: 3.7.0


[KIP-875: First-class offsets support in Kafka Connect|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect] introduced a new {{STOPPED}} state for connectors along with some REST API endpoints to retrieve and modify offsets for connectors. Currently, only connectors that already exist can be stopped and any newly created connector will always be in the {{RUNNING}} state initially. Allowing the creation of connectors in a {{STOPPED}} state will facilitate multiple new use cases. One interesting use case would be to migrate connectors from one Kafka Connect cluster to another. Individual connector migration would be useful in a number of scenarios such as breaking a large cluster into multiple smaller clusters (or vice versa), moving a connector from a cluster running in one data center to another etc. A connector migration could be achieved by using the following sequence of steps :-
 # Stop the running connector on the original Kafka Connect cluster
 # Retrieve the offsets for the connector via the {{GET /connectors/\{connector}/offsets}}  endpoint
 # Create the connector in a stopped state using the same configuration on the new Kafka Connect cluster
 # Alter the offsets for the connector on the new cluster via the {{PATCH /connectors/\{connector}/offsets}}  endpoint (using the offsets obtained from the original cluster)
 # Resume the connector on the new cluster and delete it on the original cluster

Another use case for creating connectors in a stopped state could be deploying connectors as a part of a larger data pipeline before the source / sink data system has been created or is ready for data transfer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)