You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "gaurav-narula (via GitHub)" <gi...@apache.org> on 2026/05/27 10:48:54 UTC

[PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

gaurav-narula opened a new pull request, #22384:
URL: https://github.com/apache/kafka/pull/22384

   This change adds validation for Uuids in Formatter which is used by `kafka-storage.sh` which allows users to fail fast before they inadvertently end up using an invalid cluster-id.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon merged PR #22384:
URL: https://github.com/apache/kafka/pull/22384


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3380173308


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   I revisited [KIP-78](https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id) where this was introduced and it seems that the authors never **_mandated_** the use of UUIDs:
   
   > We update KafkaServer.startup() to invoke getOrGenerateClusterId() after initZk().
   The implementation of getOrGenerateClusterId() is straightforward. It first tries to get the cluster id from /cluster/id. If the znode does not exist, it generates an id via UUID.randomUUID(), converts it to a String via URL-safe Base64 encoding and tries to create the znode. If the creation succeeds, the generated id is returned. If it fails with a ZkNodeExistsException, then another broker won the race, so we can just retrieve the cluster id from the expected znode path and return it. The returned cluster id will be passed to registered metric reporters who implement ClusterResourceListener and it will be logged at info level.
   
   IIUC, that leads to two situations:
   
   1. Operators had the brokers create the ZNode themselves, in which case it will be a base64 url encoded uuid
   2. Operators pre-created the ZNode themselves, in which case it could be populated with anything and the brokers [do not enforce](https://github.com/apache/kafka/pull/1830/changes#diff-3638ff970bc6766f9e570f76a6440966930b990af33e53e5b3db9d65f90264e9R203) any validation on the value
   
   (2) is unfortunate because the KIP talks about restricting the set of valid characters
   
   > We propose adding the cluster_id to the Metadata response, storing it in ZooKeeper, exposing it via metrics, to serializers and client interceptors. The cluster_id should be unique,  immutable, auto-generated and it may be used in situations where the allowed character set is restricted. Given this, we limit the allowed characters to the following regular expression [a-zA-Z0-9_\-]+, which corresponds to the characters used by the URL-safe Base64 variant.
   
   With KRaft, `kafka-storage.sh` doubles down by not performing any validation while formatting. Rethinking about this, the validation in this PR does break compatibility and should
   
   (a) Either be reversed or
   (b) Provide an override that disables validation check for users looking to reformat existing clusters
   
   I also feel `SharedServer#clusterId` can do with some documentation as to why its type is a String and it's unsafe to parse it as a UUID and will help reviewers and contributors alike.
   
   WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on PR #22384:
URL: https://github.com/apache/kafka/pull/22384#issuecomment-4646407312

   Will merge it after CI completes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3390808440


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   I'm going to revert the commit: 6944bce421d818e791dcca0a4806aeade1147ca9



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3324292695


##########
metadata/src/test/java/org/apache/kafka/metadata/storage/FormatterTest.java:
##########
@@ -587,4 +587,18 @@ public void testFormatWithNoInitialControllers() throws Exception {
             assertNotNull(logDirProps1);
         }
     }
+
+    @ParameterizedTest
+    @ValueSource(strings = {"unrvTtQISjar0JUWGU/8Pg", "igNUVIdeSPO5JCZYFhOh7Q==", "AAAAAAAAAAAAAAAAAAAAAA", "AAAAAAAAAAAAAAAAAAAAAQ"})
+    public void testFormatWithInvalidClusterId(String clusterId) throws Exception {
+        try (TestEnv testEnv = new TestEnv(2)) {
+            FormatterContext formatter1 = testEnv.newFormatter();
+            formatter1.formatter.setClusterId(clusterId);
+            String expectedPrefix = "The specified cluster id, " + clusterId;
+            assertEquals(expectedPrefix,
+                    assertThrows(FormatterException.class,
+                            formatter1.formatter::run).
+                            getMessage().substring(0, expectedPrefix.length()));

Review Comment:
   I'm indifferent here - I followed the convention used in another test in the same class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3324220555


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");
+            }
+            Uuid uuid = Uuid.fromString(clusterId);
+            if (Uuid.RESERVED.contains(uuid)) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " is reserved");
+            }

Review Comment:
   Good point. Let's add a comment about this for future reference.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gnarula (via GitHub)" <gi...@apache.org>.
gnarula commented on PR #22384:
URL: https://github.com/apache/kafka/pull/22384#issuecomment-4564675225

   CC: @showuon @fvaleri 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3379769424


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");

Review Comment:
   If we care about the equality, should we simply verify it via the `toString` instead? For example:
   
   ```java
   Uuid uuid = Uuid.fromString(clusterId);
   if (!clusterId.equals(uuid.toString())) {
       throw new FormatterException("Invalid cluster id format");
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3384495298


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   OK, let's revert it. Could you open a PR for the revert? Thanks @gaurav-narula @chia7712 .
   
   > What about adding a flag to enforce UUID validation? That way you get the desired behavior as opt-in and no existing procedure would be broken.
   
   I think the desired behavior should follow the KIP, which is no validation while formatting, unless we open another KIP to change that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on PR #22384:
URL: https://github.com/apache/kafka/pull/22384#issuecomment-4649622006

   > Will merge it after CI completes.
   
   Updated ReconfigurableQuorumIntegrationTest in [f48bb2d](https://github.com/apache/kafka/pull/22384/commits/f48bb2d74719329d3e0a3ff3bf5352b56c82e60d) as they failed due to the new validation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3323541536


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");
+            }
+            Uuid uuid = Uuid.fromString(clusterId);
+            if (Uuid.RESERVED.contains(uuid)) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " is reserved");
+            }

Review Comment:
   I think these are cosmetic improvemenst that've been added over time but have no bearing on correctness. Failing validation on them would hinder migration from older clusters where `-` was allowed.
   
   The motivation for avoiding `-` in the beginning was to avoid shell escaping issues when passing cluster id in CLI tools. https://issues.apache.org/jira/browse/KAFKA-13741
   
   Lately, https://issues.apache.org/jira/browse/KAFKA-20072 avoided `-` altogether to allow easier copy-pasting. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3390867422


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   Raised #22539



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "showuon (via GitHub)" <gi...@apache.org>.
showuon commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3323506973


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");
+            }
+            Uuid uuid = Uuid.fromString(clusterId);
+            if (Uuid.RESERVED.contains(uuid)) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " is reserved");
+            }

Review Comment:
   Let's also validate the starting `-` sign.



##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");

Review Comment:
   Our [randomUuid method](https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/Uuid.java#L71-L76) is possible to output the uuid contains `=` sign:
   >  This will not generate a UUID equal to 0, 1, or one whose string representation starts with a dash ("-")
   
   So is this the expected validation?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3323756586


##########
metadata/src/test/java/org/apache/kafka/metadata/storage/FormatterTest.java:
##########
@@ -587,4 +587,18 @@ public void testFormatWithNoInitialControllers() throws Exception {
             assertNotNull(logDirProps1);
         }
     }
+
+    @ParameterizedTest
+    @ValueSource(strings = {"unrvTtQISjar0JUWGU/8Pg", "igNUVIdeSPO5JCZYFhOh7Q==", "AAAAAAAAAAAAAAAAAAAAAA", "AAAAAAAAAAAAAAAAAAAAAQ"})
+    public void testFormatWithInvalidClusterId(String clusterId) throws Exception {

Review Comment:
   There's no corresponding positive test showing that a valid CID passes validation. This would guard against the validation being too aggressive.



##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");

Review Comment:
   ```suggestion
                   throw new FormatterException("The specified cluster id, " + clusterId + " is invalid: contains padding");
   ```



##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");

Review Comment:
   I think this actually covers a real gap. The one in which a CID like `igNUVIdeSPO5JCZYFhOh7Q==` would pass the validation, but would be returned as `igNUVIdeSPO5JCZYFhOh7Q` by Uuid.toString(). I would just move it outside the try-catch block as it throws FormatterException directly and would bypass the catch anyway.



##########
metadata/src/test/java/org/apache/kafka/metadata/storage/FormatterTest.java:
##########
@@ -587,4 +587,18 @@ public void testFormatWithNoInitialControllers() throws Exception {
             assertNotNull(logDirProps1);
         }
     }
+
+    @ParameterizedTest
+    @ValueSource(strings = {"unrvTtQISjar0JUWGU/8Pg", "igNUVIdeSPO5JCZYFhOh7Q==", "AAAAAAAAAAAAAAAAAAAAAA", "AAAAAAAAAAAAAAAAAAAAAQ"})
+    public void testFormatWithInvalidClusterId(String clusterId) throws Exception {
+        try (TestEnv testEnv = new TestEnv(2)) {
+            FormatterContext formatter1 = testEnv.newFormatter();
+            formatter1.formatter.setClusterId(clusterId);
+            String expectedPrefix = "The specified cluster id, " + clusterId;
+            assertEquals(expectedPrefix,
+                    assertThrows(FormatterException.class,
+                            formatter1.formatter::run).
+                            getMessage().substring(0, expectedPrefix.length()));

Review Comment:
   Wouldn't be easier to rewrite like:  
   
   assertTrue(message.startsWith(expectedPrefix))
   



##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");
+            }
+            Uuid uuid = Uuid.fromString(clusterId);
+            if (Uuid.RESERVED.contains(uuid)) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " is reserved");
+            }

Review Comment:
   Good point. Let's add comment about this for future reference.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3390827336


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   Sounds good @chia7712 . I'll take up drafting a KIP for proposing a new validation flag to `kafka-storage.sh` and limiting the check to the regex above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3323541536


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");
+            }
+            Uuid uuid = Uuid.fromString(clusterId);
+            if (Uuid.RESERVED.contains(uuid)) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " is reserved");
+            }

Review Comment:
   I think these are cosmetic improvement that's been added over time but have no bearing on correctness. Failing validation on them would hinder migration from older clusters where `-` was allowed.
   
   The motivation for avoiding `-` in the beginning was to avoid shell escaping issues when passing cluster id in CLI tools. https://issues.apache.org/jira/browse/KAFKA-13741
   
   Lately, https://issues.apache.org/jira/browse/KAFKA-20072 avoided `-` altogether to allow easier copy-pasting. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "gaurav-narula (via GitHub)" <gi...@apache.org>.
gaurav-narula commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3323520265


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -236,6 +236,17 @@ public void run() throws Exception {
         if (clusterId == null) {
             throw new FormatterException("You must specify the cluster id.");
         }
+        try {
+            if (clusterId.contains("=")) {
+                throw new FormatterException("The specified cluster id, " + clusterId + " contains padding and is invalid");

Review Comment:
   I don't think it is possible because `Uuid#toString` does not use padding and `=` sign can only be present when padding is enabled



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3390844875


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   ugh, we can't push commit to trunk directly :laughing: 
   
   @gaurav-narula Do you have free time to open PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3379906233


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   Does this change mean it breaks something? Users can't use the human-readable cluster id anymore?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3380564544


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   What about adding a flag to enforce UUID validation? That way you get the desired behavior as opt-in and no existing procedure would be broken.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3385850084


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   > I think the desired behavior should follow the KIP, which is no validation while formatting, unless we open another KIP to change that.
   
   Maybe I expressed that poorly, but it is what I was saying. Follow the KIP as default and add a flag to enable UUID validation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3387400823


##########
server/src/test/java/org/apache/kafka/server/ReconfigurableQuorumIntegrationTest.java:
##########
@@ -255,7 +255,7 @@ public void testNewVoterAutoRemovesAndAdds() throws Exception {
     @Test
     public void testRemoveAndAddVoterWithValidClusterId() throws Exception {
         final var nodes = new TestKitNodes.Builder()
-            .setClusterId("test-cluster")
+            .setClusterId("5EqhrOPYSkaSsXk4RYkNow")

Review Comment:
   If the "=" character is the main issue, we could file a KIP for it. KIP-78 explicitly states that the valid characters are `[a-zA-Z0-9_\-]+`, but it seems we never implemented that constraint :cry: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20617: add validation for cluster-id in Formatter [kafka]

Posted by "fvaleri (via GitHub)" <gi...@apache.org>.
fvaleri commented on code in PR #22384:
URL: https://github.com/apache/kafka/pull/22384#discussion_r3324627015


##########
metadata/src/main/java/org/apache/kafka/metadata/storage/Formatter.java:
##########
@@ -243,13 +229,33 @@ public BootstrapMetadata bootstrapMetadata() {
         return bootstrapMetadata;
     }
 
+    /**
+     * Validates the correctness of the given cluster id. A valid cluster id is a base64, urlencoded, no padding
+     * representation of a {@link Uuid}. These checks do not validate the absence of <code>-</code> character as
+     * {@link Uuid#randomUuid()} avoids them only for convenience reasons.

Review Comment:
   I would explicitly mention support for already generated CIDs. Basically it's for historical reasons.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org