[GitHub] [kafka] yashmayya commented on a diff in pull request #13276: KAFKA-14732: Use an exponential backoff retry mechanism while reconfiguring connector tasks

["yashmayya (via GitHub)" <gi...@apache.org>]

yashmayya commented on code in PR #13276:
URL: https://github.com/apache/kafka/pull/13276#discussion_r1112987230


##########
connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java:
##########
@@ -1870,7 +1880,16 @@ private Callable<Void> getConnectorStoppingCallable(final String connectorName)
         };
     }
 
-    private void reconfigureConnectorTasksWithRetry(long initialRequestTime, final String connName) {
+    /**
+     * Request task configs from the connector and write them to the config storage in case the configs are detected to
+     * have changed. This method retries infinitely in case of any errors.

Review Comment:
   Hm, that's an interesting idea and I don't see the harm in limiting the number of retries to some reasonable value and then marking the connector as failed after that (we could include the last exception's trace in the connector's status).
   
   > Is it possible to somehow bubble up errors as part of connector (not task) status?
   
   The `AbstractHerder` (`DistributedHerder`'s parent class) implements the `ConnectorStatus.Listener` interface and so we should be able to update the connector's status to failed via https://github.com/apache/kafka/blob/81b3b2fb3399ab2784eb8158564fc1c9a1299a8d/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L191-L195
   
   For instance, [here](https://github.com/apache/kafka/blob/81b3b2fb3399ab2784eb8158564fc1c9a1299a8d/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1856) we use the `onFailure` hook to update a connector's status as failed if there is an exception thrown during startup. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org