You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "smjn (via GitHub)" <gi...@apache.org> on 2026/05/14 10:36:09 UTC

[PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

smjn opened a new pull request, #22284:
URL: https://github.com/apache/kafka/pull/22284

   * The `ShareGroupDLQStateManager` is responsible for sending various RPCs to support the DLQ operation.
   * In this PR, we have added code to auto create the DLQ topic, if appropriate configs are valid.
   * Tests will be added when PRODUCE logic is integrated as well. The component is not yet stitched with `SharePartition`.
   * Minor change: changed factory method in `DefaultShareGroupDLQManager` to static.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on PR #22284:
URL: https://github.com/apache/kafka/pull/22284#issuecomment-4449919401

   ## Manual testing
   ### Before DLQ logic
   ```
   $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list                                                   
   __consumer_offsets
   t1
   
   $ bin/kafka-features.sh --bootstrap-server localhost:9092 upgrade --feature share.version=2
   share.version was upgraded to 2.
   
   
   $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "share.auto.offset.reset=earliest" --entity-type groups --entity-name gs1
   Completed updating config for group gs1.
   
   
   $ bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --add-config "errors.deadletterqueue.topic.name=dlq.gs1dlqtopic" --entity-type groups --entity-name gs1
   ```
   
   ### After logic kicks in 
   ```
   $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list
   __consumer_offsets
   __share_group_state
   dlq.gs1dlqtopic
   t1
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield commented on code in PR #22284:
URL: https://github.com/apache/kafka/pull/22284#discussion_r3242387579


##########
server-common/src/main/java/org/apache/kafka/server/share/dlq/ShareGroupDLQStateManager.java:
##########
@@ -165,53 +236,104 @@ public Optional<Throwable> validateDlqTopic() {
             });
         }
 
-        public ShareGroupDLQRecordParameter recordParam() {
-            return param;
-        }
-
         public boolean dlqTopicExists() {
             Optional<String> shareGroupDlqTopic = cacheHelper.shareGroupDlqTopic(param.groupId());
             return shareGroupDlqTopic.filter(cacheHelper::containsTopic).isPresent();
         }
-    }
 
-    private class ProduceRequestHandler extends ShareGroupDLQStateManagerHandler {
-        private final CompletableFuture<Void> result;
-        private static final Logger LOG = LoggerFactory.getLogger(ShareGroupDLQStateManager.ProduceRequestHandler.class);
-
-        public ProduceRequestHandler(ShareGroupDLQRecordParameter param, CompletableFuture<Void> result) {
-            super(param);
-            this.result = result;
+        public String dlqTopicName() {

Review Comment:
   Might it be a good idea to return `Optional<String>` here so that you don't inadvertently end up auto-creating a topic called `"UNKNOWN"` due to some code quirk?



##########
server-common/src/main/java/org/apache/kafka/server/share/dlq/ShareGroupDLQStateManager.java:
##########
@@ -165,53 +236,104 @@ public Optional<Throwable> validateDlqTopic() {
             });
         }
 
-        public ShareGroupDLQRecordParameter recordParam() {
-            return param;
-        }
-
         public boolean dlqTopicExists() {
             Optional<String> shareGroupDlqTopic = cacheHelper.shareGroupDlqTopic(param.groupId());
             return shareGroupDlqTopic.filter(cacheHelper::containsTopic).isPresent();
         }
-    }
 
-    private class ProduceRequestHandler extends ShareGroupDLQStateManagerHandler {
-        private final CompletableFuture<Void> result;
-        private static final Logger LOG = LoggerFactory.getLogger(ShareGroupDLQStateManager.ProduceRequestHandler.class);
-
-        public ProduceRequestHandler(ShareGroupDLQRecordParameter param, CompletableFuture<Void> result) {
-            super(param);
-            this.result = result;
+        public String dlqTopicName() {
+            return cacheHelper.shareGroupDlqTopic(param.groupId()).orElse("UNKNOWN");
         }
 
-        @Override
-        protected AbstractRequest.Builder<? extends AbstractRequest> requestBuilder() {
-            return null;
-        }
-
-        @Override
-        protected CompletableFuture<? extends AbstractResponse> result() {
-            return CompletableFuture.completedFuture(null);
-        }
+        // Visibility for testing
+        Optional<Errors> checkResponseError(ClientResponse response) {
+            if (response.hasResponse()) {
+                return Optional.empty();
+            }
 
-        @Override
-        protected String name() {
-            return "ProduceRequestHandler";
+            LOG.debug("Response for RPC {} with DLQ topic {} is invalid - {}", name(), dlqTopicName(), response);
+
+            if (response.authenticationException() != null) {
+                LOG.error("Authentication exception", response.authenticationException());
+                Errors error = Errors.forException(response.authenticationException());
+                return Optional.of(error);
+            } else if (response.versionMismatch() != null) {
+                LOG.error("Version mismatch exception", response.versionMismatch());
+                Errors error = Errors.forException(response.versionMismatch());
+                return Optional.of(error);
+            } else if (response.wasDisconnected()) {    // Retriable
+                return Optional.of(Errors.NETWORK_EXCEPTION);
+            } else if (response.wasTimedOut()) {    // Retriable
+                LOG.debug("Response for RPC {} with DLQ topic {} timed out - {}.", name(), dlqTopicName(), response);
+                return Optional.of(Errors.REQUEST_TIMED_OUT);
+            } else {
+                return Optional.of(Errors.UNKNOWN_SERVER_ERROR);
+            }
         }
 
-        @Override
-        protected void createTopicErrorResponse(Exception exception) {
-            this.result.completeExceptionally(exception);
-        }
+        private void handleCreateTopicsResponse(ClientResponse response) {
+            LOG.debug("Received CreateTopicsResponse {}", response);
+            createTopicsBackoff.incrementAttempt();
+            Errors clientResponseError = checkResponseError(response).orElse(Errors.NONE);
+            String clientResponseErrorMessage = clientResponseError.message();
+            String dlqTopicName = dlqTopicName();
+
+            switch (clientResponseError) {
+                case NONE:
+                    // Topic has been created
+                    CreateTopicsResponse createTopicsResponse = ((CreateTopicsResponse) response.responseBody());
+                    Optional<CreateTopicsResponseData.CreatableTopicResult> topicResultOpt = createTopicsResponse.data().topics().stream().findFirst();
+                    if (topicResultOpt.isEmpty() || !topicResultOpt.get().name().equals(dlqTopicName)) {
+                        LOG.error("DLQ topic not found in create topic response {}.", dlqTopicName);
+                        requestErrorResponse(Errors.UNKNOWN_TOPIC_OR_PARTITION.exception());

Review Comment:
   This would be very strange. It would mean that you said to create one topic with a particular name and the response was not empty but the topic name did not match. `INTERNAL_SERVER_ERROR`?



##########
server-common/src/main/java/org/apache/kafka/server/share/dlq/ShareGroupDLQStateManager.java:
##########
@@ -165,53 +236,104 @@ public Optional<Throwable> validateDlqTopic() {
             });
         }
 
-        public ShareGroupDLQRecordParameter recordParam() {
-            return param;
-        }
-
         public boolean dlqTopicExists() {
             Optional<String> shareGroupDlqTopic = cacheHelper.shareGroupDlqTopic(param.groupId());
             return shareGroupDlqTopic.filter(cacheHelper::containsTopic).isPresent();
         }
-    }
 
-    private class ProduceRequestHandler extends ShareGroupDLQStateManagerHandler {
-        private final CompletableFuture<Void> result;
-        private static final Logger LOG = LoggerFactory.getLogger(ShareGroupDLQStateManager.ProduceRequestHandler.class);
-
-        public ProduceRequestHandler(ShareGroupDLQRecordParameter param, CompletableFuture<Void> result) {
-            super(param);
-            this.result = result;
+        public String dlqTopicName() {
+            return cacheHelper.shareGroupDlqTopic(param.groupId()).orElse("UNKNOWN");
         }
 
-        @Override
-        protected AbstractRequest.Builder<? extends AbstractRequest> requestBuilder() {
-            return null;
-        }
-
-        @Override
-        protected CompletableFuture<? extends AbstractResponse> result() {
-            return CompletableFuture.completedFuture(null);
-        }
+        // Visibility for testing
+        Optional<Errors> checkResponseError(ClientResponse response) {
+            if (response.hasResponse()) {
+                return Optional.empty();
+            }
 
-        @Override
-        protected String name() {
-            return "ProduceRequestHandler";
+            LOG.debug("Response for RPC {} with DLQ topic {} is invalid - {}", name(), dlqTopicName(), response);
+
+            if (response.authenticationException() != null) {
+                LOG.error("Authentication exception", response.authenticationException());
+                Errors error = Errors.forException(response.authenticationException());
+                return Optional.of(error);
+            } else if (response.versionMismatch() != null) {
+                LOG.error("Version mismatch exception", response.versionMismatch());
+                Errors error = Errors.forException(response.versionMismatch());
+                return Optional.of(error);
+            } else if (response.wasDisconnected()) {    // Retriable
+                return Optional.of(Errors.NETWORK_EXCEPTION);
+            } else if (response.wasTimedOut()) {    // Retriable
+                LOG.debug("Response for RPC {} with DLQ topic {} timed out - {}.", name(), dlqTopicName(), response);
+                return Optional.of(Errors.REQUEST_TIMED_OUT);
+            } else {
+                return Optional.of(Errors.UNKNOWN_SERVER_ERROR);
+            }
         }
 
-        @Override
-        protected void createTopicErrorResponse(Exception exception) {
-            this.result.completeExceptionally(exception);
-        }
+        private void handleCreateTopicsResponse(ClientResponse response) {
+            LOG.debug("Received CreateTopicsResponse {}", response);
+            createTopicsBackoff.incrementAttempt();
+            Errors clientResponseError = checkResponseError(response).orElse(Errors.NONE);
+            String clientResponseErrorMessage = clientResponseError.message();
+            String dlqTopicName = dlqTopicName();
+
+            switch (clientResponseError) {
+                case NONE:
+                    // Topic has been created
+                    CreateTopicsResponse createTopicsResponse = ((CreateTopicsResponse) response.responseBody());
+                    Optional<CreateTopicsResponseData.CreatableTopicResult> topicResultOpt = createTopicsResponse.data().topics().stream().findFirst();
+                    if (topicResultOpt.isEmpty() || !topicResultOpt.get().name().equals(dlqTopicName)) {
+                        LOG.error("DLQ topic not found in create topic response {}.", dlqTopicName);
+                        requestErrorResponse(Errors.UNKNOWN_TOPIC_OR_PARTITION.exception());
+                        break;
+                    }
 
-        @Override
-        public void onComplete(ClientResponse response) {
+                    CreateTopicsResponseData.CreatableTopicResult topicResult = topicResultOpt.get();
+                    Errors error = Errors.forCode(topicResult.errorCode());
+                    String errorMessage = topicResult.errorMessage();
+                    switch (error) {
+                        case NONE:
+                            // Replace with enqueue post PRODUCE implementation
+                            this.result.complete(null);
+                            break;
+
+                        case THROTTLING_QUOTA_EXCEEDED:
+                            LOG.debug("Received retriable error in create DLQ topic response for {} using dlq topic {}: {}", name(), dlqTopicName, errorMessage);

Review Comment:
   such a nit: "DLQ" and "dlq" in the same line??? 😆 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #22284:
URL: https://github.com/apache/kafka/pull/22284#discussion_r3242658002


##########
server-common/src/main/java/org/apache/kafka/server/share/dlq/ShareGroupDLQStateManager.java:
##########
@@ -165,53 +236,104 @@ public Optional<Throwable> validateDlqTopic() {
             });
         }
 
-        public ShareGroupDLQRecordParameter recordParam() {
-            return param;
-        }
-
         public boolean dlqTopicExists() {
             Optional<String> shareGroupDlqTopic = cacheHelper.shareGroupDlqTopic(param.groupId());
             return shareGroupDlqTopic.filter(cacheHelper::containsTopic).isPresent();
         }
-    }
 
-    private class ProduceRequestHandler extends ShareGroupDLQStateManagerHandler {
-        private final CompletableFuture<Void> result;
-        private static final Logger LOG = LoggerFactory.getLogger(ShareGroupDLQStateManager.ProduceRequestHandler.class);
-
-        public ProduceRequestHandler(ShareGroupDLQRecordParameter param, CompletableFuture<Void> result) {
-            super(param);
-            this.result = result;
+        public String dlqTopicName() {

Review Comment:
   @AndrewJSchofield I will remove this method. It's sole purpose was logging related but I see that issue that it could cause.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "AndrewJSchofield (via GitHub)" <gi...@apache.org>.
AndrewJSchofield merged PR #22284:
URL: https://github.com/apache/kafka/pull/22284


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on PR #22284:
URL: https://github.com/apache/kafka/pull/22284#issuecomment-4452456589

   @AndrewJSchofield Thanks for the feedback, inc comments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20549: Add logic for create topic in DLQ state manager. [3/N] [kafka]

Posted by "smjn (via GitHub)" <gi...@apache.org>.
smjn commented on code in PR #22284:
URL: https://github.com/apache/kafka/pull/22284#discussion_r3242665395


##########
server-common/src/main/java/org/apache/kafka/server/share/dlq/ShareGroupDLQStateManager.java:
##########
@@ -165,53 +236,104 @@ public Optional<Throwable> validateDlqTopic() {
             });
         }
 
-        public ShareGroupDLQRecordParameter recordParam() {
-            return param;
-        }
-
         public boolean dlqTopicExists() {
             Optional<String> shareGroupDlqTopic = cacheHelper.shareGroupDlqTopic(param.groupId());
             return shareGroupDlqTopic.filter(cacheHelper::containsTopic).isPresent();
         }
-    }
 
-    private class ProduceRequestHandler extends ShareGroupDLQStateManagerHandler {
-        private final CompletableFuture<Void> result;
-        private static final Logger LOG = LoggerFactory.getLogger(ShareGroupDLQStateManager.ProduceRequestHandler.class);
-
-        public ProduceRequestHandler(ShareGroupDLQRecordParameter param, CompletableFuture<Void> result) {
-            super(param);
-            this.result = result;
+        public String dlqTopicName() {
+            return cacheHelper.shareGroupDlqTopic(param.groupId()).orElse("UNKNOWN");
         }
 
-        @Override
-        protected AbstractRequest.Builder<? extends AbstractRequest> requestBuilder() {
-            return null;
-        }
-
-        @Override
-        protected CompletableFuture<? extends AbstractResponse> result() {
-            return CompletableFuture.completedFuture(null);
-        }
+        // Visibility for testing
+        Optional<Errors> checkResponseError(ClientResponse response) {
+            if (response.hasResponse()) {
+                return Optional.empty();
+            }
 
-        @Override
-        protected String name() {
-            return "ProduceRequestHandler";
+            LOG.debug("Response for RPC {} with DLQ topic {} is invalid - {}", name(), dlqTopicName(), response);
+
+            if (response.authenticationException() != null) {
+                LOG.error("Authentication exception", response.authenticationException());
+                Errors error = Errors.forException(response.authenticationException());
+                return Optional.of(error);
+            } else if (response.versionMismatch() != null) {
+                LOG.error("Version mismatch exception", response.versionMismatch());
+                Errors error = Errors.forException(response.versionMismatch());
+                return Optional.of(error);
+            } else if (response.wasDisconnected()) {    // Retriable
+                return Optional.of(Errors.NETWORK_EXCEPTION);
+            } else if (response.wasTimedOut()) {    // Retriable
+                LOG.debug("Response for RPC {} with DLQ topic {} timed out - {}.", name(), dlqTopicName(), response);
+                return Optional.of(Errors.REQUEST_TIMED_OUT);
+            } else {
+                return Optional.of(Errors.UNKNOWN_SERVER_ERROR);
+            }
         }
 
-        @Override
-        protected void createTopicErrorResponse(Exception exception) {
-            this.result.completeExceptionally(exception);
-        }
+        private void handleCreateTopicsResponse(ClientResponse response) {
+            LOG.debug("Received CreateTopicsResponse {}", response);
+            createTopicsBackoff.incrementAttempt();
+            Errors clientResponseError = checkResponseError(response).orElse(Errors.NONE);
+            String clientResponseErrorMessage = clientResponseError.message();
+            String dlqTopicName = dlqTopicName();
+
+            switch (clientResponseError) {
+                case NONE:
+                    // Topic has been created
+                    CreateTopicsResponse createTopicsResponse = ((CreateTopicsResponse) response.responseBody());
+                    Optional<CreateTopicsResponseData.CreatableTopicResult> topicResultOpt = createTopicsResponse.data().topics().stream().findFirst();
+                    if (topicResultOpt.isEmpty() || !topicResultOpt.get().name().equals(dlqTopicName)) {
+                        LOG.error("DLQ topic not found in create topic response {}.", dlqTopicName);
+                        requestErrorResponse(Errors.UNKNOWN_TOPIC_OR_PARTITION.exception());

Review Comment:
   Ok, I will remove this check for now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org