You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mannoopj (via GitHub)" <gi...@apache.org> on 2023/09/14 17:28:50 UTC

[GitHub] [kafka] mannoopj opened a new pull request, #14389: Metadata schema checker

mannoopj opened a new pull request, #14389:
URL: https://github.com/apache/kafka/pull/14389

   Check schema changes to ensure compatibility is not broken. Done through the use of comparing new json file to historical json file. Assume historical schemas are recursively compatible.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456441308


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {

Review Comment:
   It's not enough for the line to contain `//`. It needs to start with `//` (after whitespace has been trimmed)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362702777


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));

Review Comment:
   Seems like "api keys didn't match" is the problem... we don't know which one is wrong



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1364087940


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));
+        newLatestVersion = Character.getNumericValue(newValidVersions.charAt(newValidVersions.length() - 2));
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest versions, can at most be one higher than the previous");
+        }
+        oldFirstVersion = Character.getNumericValue(oldValidVersions.charAt(1));
+        newFirstVersion = Character.getNumericValue(newValidVersions.charAt(1));
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("versions")));
+
+        if (!Arrays.equals(oldVersion, newVersion)) {
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                int cutoff = Integer.parseInt(oldVersion[1]);
+                if (cutoff > Integer.parseInt(newVersion[0])) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new schema cannot reopen already closed field");
+                }
+            } else if (oldVersion.length == 2 && newVersion.length == 2) {
+                throw new IllegalStateException("cannot changed already closed field");
+            } else if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("cannot change lower end of ");
+                }
+                int cutoffVersion = Integer.parseInt(newVersion[1]);
+                if (cutoffVersion != newLatestVersion && cutoffVersion + 1 != newLatestVersion) {
+                    throw new IllegalStateException("Invalid closing version for field");
+                }
+            } else if (oldVersion.length == 1 && newVersion.length == 1) {
+                int oldInt = Integer.parseInt(oldVersion[0]);
+                int newInt = Integer.parseInt(newVersion[0]);
+                if (oldInt < newInt) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new field needs to be on a new version");
+                }
+            }
+        }
+        return false;
+    }
+
+    private static void parseNullableVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("nullableVersions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("nullableVersions")));
+        if (nodeOrig.has("nullableVersions")) {
+            if (!nodeNew.has("nullableVersions")) {
+                throw new IllegalStateException("field is missing nullable information");
+            }
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                throw new IllegalStateException("cannot make field nullable after closing nullable versions");
+            }
+            if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("invalid closing version for nullable versions");
+                }
+                if (Integer.parseInt(newVersion[1]) != newLatestVersion) {
+                    throw new IllegalStateException("incorrect closing version for nullable versions");
+                }
+            }
+
+        } else if (nodeNew.has("nullableVersions")) {
+            if (Integer.parseInt(newVersion[0]) != newLatestVersion) {
+                throw new IllegalStateException("invalid version for new nullable version");
+            }
+        }
+    }
+
+    private static boolean checkTaggedFieldIfTag(JsonNode origNode, JsonNode newNode) {
+        if (origNode.has("tag")) {
+            if (!newNode.has("tag")) {
+                parseNewField(newNode);
+                return true;
+            }
+            if (!newNode.has("taggedVersions")) {
+                throw new IllegalStateException("new schema tagged field is missing tagged version");
+            }
+
+
+            if (latestTag + 1 != Integer.parseInt(newNode.get("tag").asText())) {
+                throw new IllegalStateException(" tag from new schema not in numeric order, " + newNode.get("tag").asText()
+                                           + " should be "  + (latestTag + 1));
+            }
+            latestTag = Integer.parseInt(newNode.get("tag").asText());
+            String[] x = cleanUpVersionStrings(origNode.get("taggedVersions").asText());
+            latestTagVersion = Integer.parseInt(cleanUpVersionStrings(origNode.get("taggedVersions").asText())[0]);
+        } else {
+            if (newNode.has("tag")) {
+                parseNewField(newNode);
+            }
+        }
+        return false;
+    }
+
+    private static void parseNewField(JsonNode node) {
+        String[] versions = cleanUpVersionStrings(String.valueOf(node.get("versions")));
+        if (oldLatestVersion + 1 != newLatestVersion && !node.has("tag")) {
+            throw new IllegalStateException("New schemas with new fields need to be on the next version iteration");
+        }
+        if (newLatestVersion != Integer.parseInt(versions[0]) && !node.has("tag")) {
+            throw new IllegalStateException("new field version not correct");
+        }
+        if (!node.has("type") && !node.has("fields")) {
+            throw new IllegalStateException("new field requires a type if it doesn't have fields node ");
+        }
+
+        if (Integer.parseInt(versions[0]) != (oldLatestVersion + 1) && !node.has("tag")) {
+            throw new IllegalStateException("New field must be on next version");
+        }
+        if (node.has("tag")) {
+            if (!node.has("taggedVersions")) {
+                throw new IllegalStateException("new tagged field is missing tagged versions");
+            }
+            String[] taggedVersions = cleanUpVersionStrings(String.valueOf(node.get("taggedVersions")));
+
+            if (Integer.parseInt(taggedVersions[0]) != latestTagVersion + 1) {
+                throw new IllegalStateException("taggedVersion incorrect for new tagged field");
+            }
+            if (Integer.parseInt(versions[0]) < oldLatestVersion ||
+                Integer.parseInt(versions[0]) > newLatestVersion) {
+                throw new IllegalStateException("Invalid version for new tagged field");
+            }
+        }
+    }
+
+    private static String[] cleanUpVersionStrings(String string) {
+        String[] cleanedInts;
+        if (string.charAt(0) == '"') {

Review Comment:
   Yes but only for some. If i remember correctly versions and nullableVersions return a string contained by " characters whereas taggedVersions is not contained. Not sure why so I just worked around it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362702954


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));

Review Comment:
   seems like "record types didn't match" is the problem ... we don't know which one is wrong



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362700793


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;

Review Comment:
   jsonSchema seems like the wrong variable name ... more like file name?



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {

Review Comment:
   where does the number 15 come from?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456447370


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));

Review Comment:
   you don't need the slash here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456440526


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {

Review Comment:
   Please change this not to use a raw 15, but to use an appropriate constant.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #14389:
URL: https://github.com/apache/kafka/pull/14389#issuecomment-2483896791

   Duplicate of #15995


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #14389:
URL: https://github.com/apache/kafka/pull/14389#issuecomment-2274886266

   This PR is being marked as stale since it has not had any activity in 90 days. If you would like to keep this PR alive, please ask a committer for review. If the PR has  merge conflicts, please update it with the latest from trunk (or appropriate release branch) <p> If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1364322430


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {

Review Comment:
   *At least 15 lines of comments, I've noticed some with more



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362703702


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));

Review Comment:
   this is incorrect. versions can take two ascii characters. (for example, "10" is two characters.)



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));
+        newLatestVersion = Character.getNumericValue(newValidVersions.charAt(newValidVersions.length() - 2));

Review Comment:
   this is incorrect. versions can take two ascii characters. (for example, "10" is two characters.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362702312


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {

Review Comment:
   again where does 15 come from? length of git hash when expressed as ascii text perhaps?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362699472


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;

Review Comment:
   we don't do star imports (I thought checkstyle would catch this, but maybe not?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456448973


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {

Review Comment:
   this shouldn't be needed. just skip comments.



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + fileName);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {

Review Comment:
   It's not enough for the line to contain //. It needs to start with // (after whitespace has been trimmed)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456442184


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();

Review Comment:
   please change this to use something like `Paths.get` so that it's not dependent on the system path separator (which is different on Windows)



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;

Review Comment:
   please change this to use something like Paths.get so that it's not dependent on the system path separator (which is different on Windows)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456451709


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + fileName);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, Original api key: " + edited.get("apiKey") +
+                                            ", New api key: " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, Original type: " + edited.get("type") +
+                                            ", New Type: " + original.get("type"));
+        }
+        String[] oldValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        String[] newValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        oldFirstVersion = Integer.parseInt(oldValidVersions[0]);
+        newFirstVersion = Integer.parseInt(newValidVersions[0]);
+        if (oldValidVersions.length == 1) {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[0]);
+        } else {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[1]);
+        }
+        if (newValidVersions.length == 1) {
+            newLatestVersion = Integer.parseInt(newValidVersions[0]);
+        } else {
+            newLatestVersion = Integer.parseInt(newValidVersions[1]);
+        }
+
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest version, can at most be one higher than the previous. Original version: "
+                                            + oldLatestVersion + ", New Version: " + newLatestVersion);
+        }
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));

Review Comment:
   Use VersionRange here, not an array



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + fileName);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, Original api key: " + edited.get("apiKey") +
+                                            ", New api key: " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, Original type: " + edited.get("type") +
+                                            ", New Type: " + original.get("type"));
+        }
+        String[] oldValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        String[] newValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        oldFirstVersion = Integer.parseInt(oldValidVersions[0]);
+        newFirstVersion = Integer.parseInt(newValidVersions[0]);
+        if (oldValidVersions.length == 1) {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[0]);
+        } else {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[1]);
+        }
+        if (newValidVersions.length == 1) {
+            newLatestVersion = Integer.parseInt(newValidVersions[0]);
+        } else {
+            newLatestVersion = Integer.parseInt(newValidVersions[1]);
+        }
+
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest version, can at most be one higher than the previous. Original version: "
+                                            + oldLatestVersion + ", New Version: " + newLatestVersion);
+        }
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("versions")));
+
+        if (!Arrays.equals(oldVersion, newVersion)) {
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                int cutoff = Integer.parseInt(oldVersion[1]);
+                if (cutoff > Integer.parseInt(newVersion[0])) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new schema cannot reopen already closed field");
+                }
+            } else if (oldVersion.length == 2 && newVersion.length == 2) {
+                throw new IllegalStateException("cannot changed already closed field");
+            } else if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("cannot change lower end of ");
+                }
+                int cutoffVersion = Integer.parseInt(newVersion[1]);
+                if (cutoffVersion != newLatestVersion && cutoffVersion + 1 != newLatestVersion) {
+                    throw new IllegalStateException("Invalid closing version for field");
+                }
+            } else if (oldVersion.length == 1 && newVersion.length == 1) {
+                int oldInt = Integer.parseInt(oldVersion[0]);
+                int newInt = Integer.parseInt(newVersion[0]);
+                if (oldInt < newInt) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new field needs to be on a new version");
+                }
+            }
+        }
+        return false;
+    }
+
+    private static void parseNullableVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("nullableVersions")));

Review Comment:
   Use VersionRange here, not an array



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1446640522


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {

Review Comment:
   Addressed in a similar comment below



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [kafka] mannoopj commented on pull request #14389: Metadata schema checker

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on PR #14389:
URL: https://github.com/apache/kafka/pull/14389#issuecomment-1723352632

   @mumrah @ahuang98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362704344


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));
+        newLatestVersion = Character.getNumericValue(newValidVersions.charAt(newValidVersions.length() - 2));
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest versions, can at most be one higher than the previous");

Review Comment:
   Invalid latest version (singular not plural)
   
   tell us what the versions were that had the problem



##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));
+        newLatestVersion = Character.getNumericValue(newValidVersions.charAt(newValidVersions.length() - 2));
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest versions, can at most be one higher than the previous");

Review Comment:
   should be: "Invalid latest version" (singular not plural)
   
   tell us what the versions were that had the problem



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362701664


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");

Review Comment:
   I don't see why we have to catch this stack trace. Just let it propagate to the top level and then you can see the problem when running this on the command-line.
   
   If you are going to catch it, you at least need to set the exit status code to be nonzero.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1363946935


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {

Review Comment:
   Not exactly sure if its the git hash but for all the json files there are 15 lines of comments about apache licensing. So in order to just get the json I skip the first 15 lines of the file. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe closed pull request #14389: Metadata schema checker
URL: https://github.com/apache/kafka/pull/14389


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1524953366


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + fileName);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, Original api key: " + edited.get("apiKey") +
+                                            ", New api key: " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, Original type: " + edited.get("type") +
+                                            ", New Type: " + original.get("type"));
+        }
+        String[] oldValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        String[] newValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        oldFirstVersion = Integer.parseInt(oldValidVersions[0]);
+        newFirstVersion = Integer.parseInt(newValidVersions[0]);
+        if (oldValidVersions.length == 1) {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[0]);
+        } else {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[1]);
+        }
+        if (newValidVersions.length == 1) {
+            newLatestVersion = Integer.parseInt(newValidVersions[0]);
+        } else {
+            newLatestVersion = Integer.parseInt(newValidVersions[1]);
+        }
+
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest version, can at most be one higher than the previous. Original version: "
+                                            + oldLatestVersion + ", New Version: " + newLatestVersion);
+        }
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("versions")));
+
+        if (!Arrays.equals(oldVersion, newVersion)) {
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                int cutoff = Integer.parseInt(oldVersion[1]);
+                if (cutoff > Integer.parseInt(newVersion[0])) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new schema cannot reopen already closed field");
+                }
+            } else if (oldVersion.length == 2 && newVersion.length == 2) {
+                throw new IllegalStateException("cannot changed already closed field");
+            } else if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("cannot change lower end of ");
+                }
+                int cutoffVersion = Integer.parseInt(newVersion[1]);
+                if (cutoffVersion != newLatestVersion && cutoffVersion + 1 != newLatestVersion) {
+                    throw new IllegalStateException("Invalid closing version for field");
+                }
+            } else if (oldVersion.length == 1 && newVersion.length == 1) {
+                int oldInt = Integer.parseInt(oldVersion[0]);
+                int newInt = Integer.parseInt(newVersion[0]);
+                if (oldInt < newInt) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new field needs to be on a new version");
+                }
+            }
+        }
+        return false;
+    }
+
+    private static void parseNullableVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("nullableVersions")));

Review Comment:
   I used FinalizedVersionRange here instead of VersionRange since I was unable to import VersionRange. It seems like a acceptable substitute to me but if not I can change it.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362700044


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",

Review Comment:
   why are we listing these explicitly rather than just using listDir to find out what is in the directory?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1362705401


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,347 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.CheckoutCommand;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.InitCommand;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.*;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.storage.file.FileRepositoryBuilder;
+import org.eclipse.jgit.transport.UsernamePasswordCredentialsProvider;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.*;
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+
+    static String[] filesCheckMetadata = {"AccessControlEntryRecord.json", "BrokerRegistrationChangeRecord.json", "ClientQuotaRecord.json",
+            "ConfigRecord.json", "DelegationTokenRecord.json", "FeatureLevelRecord.json", "FenceBrokerRecord.json", "NoOpRecord.json",
+            "PartitionChangeRecord.json", "PartitionRecord.json", "ProducerIdsRecord.json", "RegisterBrokerRecord.json",
+            "RemoveAccessControlEntryRecord.json", "RemoveTopicRecord.json", "RemoveUserScramCredentialRecord.json", "TopicRecord.json",
+            "UnfenceBrokerRecord.json", "UnregisterBrokerRecord.json", "UserScramCredentialRecord.json", "ZkMigrationRecord.json"};
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String jsonSchema: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + jsonSchema;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    content.append(line);
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (FileNotFoundException e) {
+            System.out.println("An error occurred.");
+            e.printStackTrace();
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String jsonSchema : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + jsonSchema));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + jsonSchema);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, " + edited.get("apiKey") + " should be " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, " + edited.get("type") + " should be " + original.get("type"));
+        }
+        String oldValidVersions = String.valueOf(original.get("validVersions"));
+        String newValidVersions = String.valueOf(edited.get("validVersions"));
+        oldLatestVersion = Character.getNumericValue(oldValidVersions.charAt(oldValidVersions.length() - 2));
+        newLatestVersion = Character.getNumericValue(newValidVersions.charAt(newValidVersions.length() - 2));
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest versions, can at most be one higher than the previous");
+        }
+        oldFirstVersion = Character.getNumericValue(oldValidVersions.charAt(1));
+        newFirstVersion = Character.getNumericValue(newValidVersions.charAt(1));
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("versions")));
+
+        if (!Arrays.equals(oldVersion, newVersion)) {
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                int cutoff = Integer.parseInt(oldVersion[1]);
+                if (cutoff > Integer.parseInt(newVersion[0])) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new schema cannot reopen already closed field");
+                }
+            } else if (oldVersion.length == 2 && newVersion.length == 2) {
+                throw new IllegalStateException("cannot changed already closed field");
+            } else if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("cannot change lower end of ");
+                }
+                int cutoffVersion = Integer.parseInt(newVersion[1]);
+                if (cutoffVersion != newLatestVersion && cutoffVersion + 1 != newLatestVersion) {
+                    throw new IllegalStateException("Invalid closing version for field");
+                }
+            } else if (oldVersion.length == 1 && newVersion.length == 1) {
+                int oldInt = Integer.parseInt(oldVersion[0]);
+                int newInt = Integer.parseInt(newVersion[0]);
+                if (oldInt < newInt) {
+                    parseNewField(nodeNew);
+                    return true;
+                } else {
+                    throw new IllegalStateException("new field needs to be on a new version");
+                }
+            }
+        }
+        return false;
+    }
+
+    private static void parseNullableVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("nullableVersions")));
+        String[] newVersion = cleanUpVersionStrings(String.valueOf(nodeNew.get("nullableVersions")));
+        if (nodeOrig.has("nullableVersions")) {
+            if (!nodeNew.has("nullableVersions")) {
+                throw new IllegalStateException("field is missing nullable information");
+            }
+            if (oldVersion.length == 2 && newVersion.length == 1) {
+                throw new IllegalStateException("cannot make field nullable after closing nullable versions");
+            }
+            if (oldVersion.length == 1 && newVersion.length == 2) {
+                if (!Objects.equals(oldVersion[0], newVersion[0])) {
+                    throw new IllegalStateException("invalid closing version for nullable versions");
+                }
+                if (Integer.parseInt(newVersion[1]) != newLatestVersion) {
+                    throw new IllegalStateException("incorrect closing version for nullable versions");
+                }
+            }
+
+        } else if (nodeNew.has("nullableVersions")) {
+            if (Integer.parseInt(newVersion[0]) != newLatestVersion) {
+                throw new IllegalStateException("invalid version for new nullable version");
+            }
+        }
+    }
+
+    private static boolean checkTaggedFieldIfTag(JsonNode origNode, JsonNode newNode) {
+        if (origNode.has("tag")) {
+            if (!newNode.has("tag")) {
+                parseNewField(newNode);
+                return true;
+            }
+            if (!newNode.has("taggedVersions")) {
+                throw new IllegalStateException("new schema tagged field is missing tagged version");
+            }
+
+
+            if (latestTag + 1 != Integer.parseInt(newNode.get("tag").asText())) {
+                throw new IllegalStateException(" tag from new schema not in numeric order, " + newNode.get("tag").asText()
+                                           + " should be "  + (latestTag + 1));
+            }
+            latestTag = Integer.parseInt(newNode.get("tag").asText());
+            String[] x = cleanUpVersionStrings(origNode.get("taggedVersions").asText());
+            latestTagVersion = Integer.parseInt(cleanUpVersionStrings(origNode.get("taggedVersions").asText())[0]);
+        } else {
+            if (newNode.has("tag")) {
+                parseNewField(newNode);
+            }
+        }
+        return false;
+    }
+
+    private static void parseNewField(JsonNode node) {
+        String[] versions = cleanUpVersionStrings(String.valueOf(node.get("versions")));
+        if (oldLatestVersion + 1 != newLatestVersion && !node.has("tag")) {
+            throw new IllegalStateException("New schemas with new fields need to be on the next version iteration");
+        }
+        if (newLatestVersion != Integer.parseInt(versions[0]) && !node.has("tag")) {
+            throw new IllegalStateException("new field version not correct");
+        }
+        if (!node.has("type") && !node.has("fields")) {
+            throw new IllegalStateException("new field requires a type if it doesn't have fields node ");
+        }
+
+        if (Integer.parseInt(versions[0]) != (oldLatestVersion + 1) && !node.has("tag")) {
+            throw new IllegalStateException("New field must be on next version");
+        }
+        if (node.has("tag")) {
+            if (!node.has("taggedVersions")) {
+                throw new IllegalStateException("new tagged field is missing tagged versions");
+            }
+            String[] taggedVersions = cleanUpVersionStrings(String.valueOf(node.get("taggedVersions")));
+
+            if (Integer.parseInt(taggedVersions[0]) != latestTagVersion + 1) {
+                throw new IllegalStateException("taggedVersion incorrect for new tagged field");
+            }
+            if (Integer.parseInt(versions[0]) < oldLatestVersion ||
+                Integer.parseInt(versions[0]) > newLatestVersion) {
+                throw new IllegalStateException("Invalid version for new tagged field");
+            }
+        }
+    }
+
+    private static String[] cleanUpVersionStrings(String string) {
+        String[] cleanedInts;
+        if (string.charAt(0) == '"') {

Review Comment:
   I'm confused how a quote character could end up here, does this actually happen?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1456448643


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));

Review Comment:
   please change this to use something like Paths.get so that it's not dependent on the system path separator (which is different on Windows)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Metadata schema checker [kafka]

Posted by "mannoopj (via GitHub)" <gi...@apache.org>.
mannoopj commented on code in PR #14389:
URL: https://github.com/apache/kafka/pull/14389#discussion_r1466452243


##########
tools/src/main/java/org/apache/kafka/tools/SchemaChecker/MetadataSchemaChecker.java:
##########
@@ -0,0 +1,359 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.tools.SchemaChecker;
+
+
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.node.ArrayNode;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.ObjectLoader;
+import org.eclipse.jgit.lib.Ref;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.revwalk.RevTree;
+import org.eclipse.jgit.revwalk.RevWalk;
+import org.eclipse.jgit.treewalk.TreeWalk;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Objects;
+
+
+public class MetadataSchemaChecker {
+
+    static int latestTag = -1;
+    static int  latestTagVersion = -1;
+    static int oldLatestVersion = -1;
+    static int oldFirstVersion = -1;
+    static int newLatestVersion = -1;
+    static int newFirstVersion = -1;
+    static String[] filesCheckMetadata = new File(System.getProperty("user.dir") + "/metadata/src/main/resources/common/metadata/").list();
+    public static void main(String[] args) throws Exception {
+
+        try {
+            List<String> localContent = new ArrayList<>();
+            for(String fileName: filesCheckMetadata) {
+                final String dir = System.getProperty("user.dir");
+                String path = dir + "/metadata/src/main/resources/common/metadata/" + fileName;
+                BufferedReader reader = new BufferedReader(new FileReader(path));
+                for (int i = 0; i < 15; i++) {
+                    reader.readLine();
+                }
+                StringBuilder content = new StringBuilder();
+                boolean print = false;
+                for (String line = reader.readLine(); line != null; line = reader.readLine()) {
+                    if (line.charAt(0) == '{') {
+                        print = true;
+                    }
+                    if (print && !line.contains("//")) {
+                        content.append(line);
+                    }
+                }
+                localContent.add(content.toString());
+            }
+
+            List<String> gitContent = GetDataFromGit();
+            if (localContent.size() != gitContent.size()) {
+                throw new IllegalStateException("missing schemas");
+            }
+            for(int i = 0; i < localContent.size(); i++) {
+                if (Objects.equals(localContent.get(i), gitContent.get(i))) {
+                    continue;
+                }
+
+                ObjectMapper objectMapper = new ObjectMapper();
+                JsonNode jsonNode1 = objectMapper.readTree(gitContent.get(i));
+                JsonNode jsonNode2 = objectMapper.readTree(localContent.get(i));
+
+                checkApiTypeVersions(jsonNode1, jsonNode2);
+                parser((ArrayNode) jsonNode1.get("fields"), (ArrayNode) jsonNode2.get("fields"));
+            }
+
+        } catch (IOException e) {
+            throw new RuntimeException(e);
+        }
+    }
+
+    private static List<String> GetDataFromGit() throws IOException, GitAPIException {
+        List<String> gitSchemas = new ArrayList<>();
+
+        Git git = Git.open(new File(System.getProperty("user.dir") + "/.git"));
+        Repository repository = git.getRepository();
+        Ref head = git.getRepository().getRefDatabase().firstExactRef("refs/heads/trunk");
+
+        try (RevWalk revWalk = new RevWalk(repository)) {
+            RevCommit commit = revWalk.parseCommit(head.getObjectId());
+            RevTree tree = commit.getTree();
+            for (String fileName : filesCheckMetadata) {
+                StringBuilder stringBuilder = new StringBuilder();
+                try (TreeWalk treeWalk = new TreeWalk(repository)) {
+                    treeWalk.addTree(tree);
+                    treeWalk.setRecursive(true);
+                    treeWalk.setFilter(PathFilter.create("metadata/src/main/resources/common/metadata/" + fileName));
+                    if (!treeWalk.next()) {
+                        throw new IllegalStateException("Did not find expected file /metadata/src/main/resources/common/metadata/" + fileName);
+                    }
+                    ObjectId objectId = treeWalk.getObjectId(0);
+                    ObjectLoader loader = repository.open(objectId);
+
+                    String content = new String(loader.getBytes());
+                    String[] lines = content.split("\n");
+                    boolean print = false;
+                    for (int i = 15; i < lines.length; i++) {
+                        if (lines[i].charAt(0) == '{') {
+                            print = true;
+                        }
+                        if (print && !lines[i].contains("//")) {
+                            stringBuilder.append(lines[i]);
+                        }
+                    }
+                }
+                gitSchemas.add(stringBuilder.toString());
+            }
+            revWalk.dispose();
+        }
+        return gitSchemas;
+    }
+
+    private static void checkApiTypeVersions(JsonNode original, JsonNode edited) {
+        if (!Objects.equals(original.get("apiKey"), edited.get("apiKey"))) {
+            throw new IllegalStateException("New schema has wrong api key, Original api key: " + edited.get("apiKey") +
+                                            ", New api key: " + original.get("apiKey"));
+        }
+        if (!Objects.equals(original.get("type"), edited.get("type"))) {
+            throw new IllegalStateException("New schema has wrong record type, Original type: " + edited.get("type") +
+                                            ", New Type: " + original.get("type"));
+        }
+        String[] oldValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        String[] newValidVersions = cleanUpVersionStrings(String.valueOf(original.get("validVersions")));
+        oldFirstVersion = Integer.parseInt(oldValidVersions[0]);
+        newFirstVersion = Integer.parseInt(newValidVersions[0]);
+        if (oldValidVersions.length == 1) {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[0]);
+        } else {
+            oldLatestVersion = Integer.parseInt(oldValidVersions[1]);
+        }
+        if (newValidVersions.length == 1) {
+            newLatestVersion = Integer.parseInt(newValidVersions[0]);
+        } else {
+            newLatestVersion = Integer.parseInt(newValidVersions[1]);
+        }
+
+        if (oldLatestVersion != newLatestVersion && oldLatestVersion + 1 != newLatestVersion) {
+            throw new IllegalStateException("Invalid latest version, can at most be one higher than the previous. Original version: "
+                                            + oldLatestVersion + ", New Version: " + newLatestVersion);
+        }
+        if (oldFirstVersion != newFirstVersion) {
+            throw new IllegalStateException("cannot change lower end of valid versions");
+        }
+    }
+
+    private static void parser(ArrayNode fieldsOrig, ArrayNode fieldsNew) {
+        Iterator<JsonNode> iterNewNode = fieldsNew.iterator();
+        Iterator<JsonNode> iterOldNode = fieldsOrig.iterator();
+        boolean isNewField = false;
+        JsonNode nodeOrig = null;
+        while (iterOldNode.hasNext() || isNewField) {
+            if (!isNewField) {
+                nodeOrig = iterOldNode.next();
+            }
+            isNewField = false;
+
+            if (!iterNewNode.hasNext()) {
+                throw new IllegalStateException("New schema is missing field ");
+            }
+            JsonNode nodeNew = iterNewNode.next();
+
+            if (checkTaggedFieldIfTag(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (parseVersion(nodeOrig, nodeNew)) {
+                isNewField = true;
+                continue;
+            }
+
+            if (!Objects.equals(nodeOrig.get("type"), nodeNew.get("type"))) {
+                throw new IllegalStateException("Fields must have same type. Expected: " + nodeOrig.get("type") + " Received: " + nodeNew.get("type"));
+            }
+
+            parseNullableVersion(nodeOrig, nodeNew);
+
+
+            if (nodeOrig.has("fields")) {
+                while(!nodeNew.has("fields")) {
+                    parseNewField(nodeNew);
+                    nodeNew = iterNewNode.next();
+                }
+                parser((ArrayNode) nodeOrig.get("fields"), (ArrayNode) nodeNew.get("fields"));
+            }
+        }
+
+        while (iterNewNode.hasNext()) {
+            parseNewField(iterNewNode.next());
+        }
+    }
+
+    private static boolean parseVersion(JsonNode nodeOrig, JsonNode nodeNew) {
+        String[] oldVersion = cleanUpVersionStrings(String.valueOf(nodeOrig.get("versions")));

Review Comment:
   I used `FinalizedVersionRange` here instead of `VersionRange` since I was unable to import `VersionRange`. It seems like a acceptable substitute to me but if not I can change it. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org