[GitHub] [kafka] C0urante commented on a change in pull request #10907: KAFKA-10000: Exactly-once support for source connectors (KIP-618)

[GitBox <gi...@apache.org>]

C0urante commented on a change in pull request #10907:
URL: https://github.com/apache/kafka/pull/10907#discussion_r730964164



##########
File path: connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedConfig.java
##########
@@ -192,6 +198,19 @@
     public static final String INTER_WORKER_VERIFICATION_ALGORITHMS_DOC = "A list of permitted algorithms for verifying internal requests";
     public static final List<String> INTER_WORKER_VERIFICATION_ALGORITHMS_DEFAULT = Collections.singletonList(INTER_WORKER_SIGNATURE_ALGORITHM_DEFAULT);
 
+    public static final String EXACTLY_ONCE_SOURCE_SUPPORT_CONFIG = "exactly.once.source.support";
+    public static final String EXACTLY_ONCE_SOURCE_SUPPORT_DOC = "Whether to enable exactly-once support for source connectors in the cluster "
+            + "by writing source records and their offsets in a Kafka transaction, and by proactively fencing out old task generations before bringing up new ones. "
+            + "Note that this must be enabled on every worker in a cluster in order for exactly-once delivery to be guaranteed, "
+            + "and that some source connectors may still not be able to provide exactly-once delivery guarantees even with this support enabled. "
+            + "Permitted values are \"disabled\", \"preparing\", and \"enabled\". In order to safely enable exactly-once support for source connectors, "
+            + "all workers in the cluster must first be updated to use the \"preparing\" value for this property. "
+            + "Once this has been done, a second update of all of the workers in the cluster should be performed to change the value of this property to \"enabled\".";

Review comment:
       > It's clear to me that we should mention this, but it's not clear where we should do so. The user documentation generated from this doc string might be one spot that users will see routinely, so maybe it's a candidate. Another would be in the Kafka Connect docs about EOS for source connectors.
   
   I think both are acceptable. I'll add this to the docstring now and include it in the high-level docs when I write those as well.
   
   > BTW, can you add to this PR changes to the Kafka docs that describe this feature?
   
   Given how massive this PR is already, I'd like to do this in a follow-up, with the understanding that Kafka docs changes are a requisite for including this feature in a release.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org