You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/02/16 10:22:55 UTC

[GitHub] [kafka] dajac opened a new pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

dajac opened a new pull request #10129:
URL: https://github.com/apache/kafka/pull/10129


   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] ijuma commented on pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
ijuma commented on pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#issuecomment-781549848


   I agree that comparing the string is going to be more efficient than converting to uuid and then comparing.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r578873790



##########
File path: clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
##########
@@ -354,7 +355,8 @@
         "Requested position is not greater than or equal to zero, and less than the size of the snapshot.",
         PositionOutOfRangeException::new),
     UNKNOWN_TOPIC_ID(100, "This server does not host this topic ID.", UnknownTopicIdException::new),
-    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new);
+    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new),
+    INVALID_CLUSTER_ID(102, "The supplied cluster id is not valid.", InvalidClusterIdException::new);

Review comment:
       I couldn't find any existing `INVALID*` error code that seems to fit this case. Usually "invalid" is reserved for cases where the field is structurally invalid. For example, `INVALID_GROUP_ID` is used when the groupid is empty in APIs where we require it to be non-empty. The closest similar case is `INVALID_PRODUCER_ID_MAPPING`. 
   
   We are going to add an INCONSISTENT_TOPIC_ID in https://github.com/apache/kafka/pull/10143. Perhaps that is  enough cover here? The usage is similar: the request indicates an id which does not match the local state.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#issuecomment-781611958


   @hachikuji @ijuma Yeah, that makes sense. I have changed this.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dengziming commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
dengziming commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r587013245



##########
File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java
##########
@@ -959,6 +972,10 @@ private FetchResponseData buildEmptyFetchResponse(
     ) {
         FetchRequestData request = (FetchRequestData) requestMetadata.data;
 
+        if (!hasValidClusterId(request)) {

Review comment:
       We only validate `FetchRequest` here? should we also add validation to the other 4 rpcs?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r578873790



##########
File path: clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
##########
@@ -354,7 +355,8 @@
         "Requested position is not greater than or equal to zero, and less than the size of the snapshot.",
         PositionOutOfRangeException::new),
     UNKNOWN_TOPIC_ID(100, "This server does not host this topic ID.", UnknownTopicIdException::new),
-    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new);
+    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new),
+    INVALID_CLUSTER_ID(102, "The supplied cluster id is not valid.", InvalidClusterIdException::new);

Review comment:
       I couldn't find any existing `INVALID*` error code that seems to fit this case. Usually "invalid" is reserved for cases where the field is structurally invalid. For example, `INVALID_GROUP_ID` is used when the groupid is empty in APIs where we require it to be non-empty. The closest similar case is `INVALID_PRODUCER_ID_MAPPING`. 
   
   We are going to add an `INCONSISTENT_TOPIC_ID` in https://github.com/apache/kafka/pull/10143. Perhaps that is  enough cover here? The usage is similar: the request indicates an id which does not match the local state.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r577993815



##########
File path: clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
##########
@@ -354,7 +355,8 @@
         "Requested position is not greater than or equal to zero, and less than the size of the snapshot.",
         PositionOutOfRangeException::new),
     UNKNOWN_TOPIC_ID(100, "This server does not host this topic ID.", UnknownTopicIdException::new),
-    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new);
+    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new),
+    INVALID_CLUSTER_ID(102, "The supplied cluster id is not valid.", InvalidClusterIdException::new);

Review comment:
       I am considering if we should have a more explicit name here. Specifically we are checking that the clusterId matches. Maybe `INCONSISTENT_CLUSTER_ID` would be clearer?

##########
File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java
##########
@@ -925,6 +931,18 @@ private FetchResponseData buildEmptyFetchResponse(
         );
     }
 
+    private boolean hasValidClusterId(FetchRequestData request) {
+        // We don't enforce the cluster id if it is not provided.
+        if (request.clusterId() == null) {
+            return true;
+        }
+        try {
+            return Uuid.fromString(request.clusterId()).equals(clusterId);
+        } catch (IllegalArgumentException e) {
+            return false;

Review comment:
       Is it worth logging something here?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r578723670



##########
File path: clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
##########
@@ -354,7 +355,8 @@
         "Requested position is not greater than or equal to zero, and less than the size of the snapshot.",
         PositionOutOfRangeException::new),
     UNKNOWN_TOPIC_ID(100, "This server does not host this topic ID.", UnknownTopicIdException::new),
-    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new);
+    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new),
+    INVALID_CLUSTER_ID(102, "The supplied cluster id is not valid.", InvalidClusterIdException::new);

Review comment:
       I see your point. `INVALID` is mainly used by all the other errors so it also makes sense to follow the "naming convention". I don't feel strong either ways.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#issuecomment-781546422


   I had some offline discussion with @cmccabe and @ijuma about the clusterId type. I think it makes sense for now to keep it as a string since this is consistent with most uses in the current codebase and it is what is documented in the KIP. I do wonder if it might be more efficient to do the comparison on the string rather than converting to Uuid though.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on a change in pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on a change in pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#discussion_r578876834



##########
File path: raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java
##########
@@ -1751,6 +1768,7 @@ private FetchRequestData buildFetchRequest() {
         });
         return request
             .setMaxWaitMs(fetchMaxWaitMs)
+            .setClusterId(clusterId.toString())

Review comment:
       One small detail which is probably ok. The clusterId field in the fetch schema is not currently marked as ignorable. That should be ok since it is only used in the raft implementation which can guarantee that we will have version 12 and above. On the other hand, I don't see any harm making the field ignorable since we are accepting a null value anyway. Is it worth changing that?

##########
File path: clients/src/main/java/org/apache/kafka/common/protocol/Errors.java
##########
@@ -354,7 +355,8 @@
         "Requested position is not greater than or equal to zero, and less than the size of the snapshot.",
         PositionOutOfRangeException::new),
     UNKNOWN_TOPIC_ID(100, "This server does not host this topic ID.", UnknownTopicIdException::new),
-    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new);
+    DUPLICATE_BROKER_REGISTRATION(101, "This broker ID is already in use.", DuplicateBrokerRegistrationException::new),
+    INVALID_CLUSTER_ID(102, "The supplied cluster id is not valid.", InvalidClusterIdException::new);

Review comment:
       I couldn't find any existing `INVALID*` error code that seems to fit this case. Usually "invalid" is reserved for cases where the field is structurally invalid. For example, `INVALID_GROUP_ID` is used when the groupid is empty. The closest similar case is `INVALID_PRODUCER_ID_MAPPING`. 
   
   We are going to add an INCONSISTENT_TOPIC_ID in https://github.com/apache/kafka/pull/10143. Perhaps that is  enough cover here? The usage is similar: the request indicates an id which does not match the local state.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji commented on pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji commented on pull request #10129:
URL: https://github.com/apache/kafka/pull/10129#issuecomment-782423782


   I verified tests locally. I am going to merge to trunk and 2.8 since we seem blocked by jenkins at the moment.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] hachikuji merged pull request #10129: KAFKA-10817; Add clusterId validation to Fetch handling

Posted by GitBox <gi...@apache.org>.
hachikuji merged pull request #10129:
URL: https://github.com/apache/kafka/pull/10129


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org