You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "cmccabe (via GitHub)" <gi...@apache.org> on 2023/12/01 23:10:31 UTC

[PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

cmccabe opened a new pull request, #14899:
URL: https://github.com/apache/kafka/pull/14899

   MetadataShell should take an advisory lock on the .lock file of the directory it is reading from. Add an integration test of this functionality in MetadataShellIntegrationTest.java.
   
   Note: in build.gradle, I had to add some dependencies on server-common's test files in order to use MockFaultHandler, etc.
   
   MetadataBatchLoader.java: fix a case where a log message was incorrect.  The intention was to print the number equivalent to (offset + index).  Instead it was printing the offset, followed by the index. So if the offset was 100 and the index was 1, 1001 would be printed rather than 101.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #14899:
URL: https://github.com/apache/kafka/pull/14899#issuecomment-1836970040

   In addition to the junit tests, I also tested this manually and it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe merged PR #14899:
URL: https://github.com/apache/kafka/pull/14899


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #14899:
URL: https://github.com/apache/kafka/pull/14899#discussion_r1412657561


##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -81,6 +85,21 @@ public MetadataShell build() {
         }
     }
 
+    static FileLock takeFileLock(File directory) {
+        FileLock fileLock = new FileLock(new File(directory, ".lock"));

Review Comment:
   I suggest not creating a `.lock` file if the specified directory doesn't contain a `.lock` file.



##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -119,6 +141,7 @@ private void initializeWithRaftManager() {
     }
 
     private void initializeWithSnapshotFileReader() throws Exception {
+        this.fileLock = takeFileLock(new File(snapshotPath).getParentFile());

Review Comment:
   This doesn't look like the correct directory. Please take a look at the `LogManager` for details.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14899:
URL: https://github.com/apache/kafka/pull/14899#discussion_r1412672706


##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -81,6 +85,21 @@ public MetadataShell build() {
         }
     }
 
+    static FileLock takeFileLock(File directory) {
+        FileLock fileLock = new FileLock(new File(directory, ".lock"));

Review Comment:
   That seems like a bad idea since it could allow another server or tool to collide with what we're doing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14899:
URL: https://github.com/apache/kafka/pull/14899#discussion_r1412672706


##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -81,6 +85,21 @@ public MetadataShell build() {
         }
     }
 
+    static FileLock takeFileLock(File directory) {
+        FileLock fileLock = new FileLock(new File(directory, ".lock"));

Review Comment:
   Yeah, that might make sense for this use-case. Since we also want to be able to look at snapshots that aren't inside log dirs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #14899:
URL: https://github.com/apache/kafka/pull/14899#discussion_r1412689568


##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -119,6 +141,7 @@ private void initializeWithRaftManager() {
     }
 
     private void initializeWithSnapshotFileReader() throws Exception {
+        this.fileLock = takeFileLock(new File(snapshotPath).getParentFile());

Review Comment:
   Fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #14899:
URL: https://github.com/apache/kafka/pull/14899#issuecomment-1844406974

   Jenkins had an internal issue. Re-pushing to get a clean build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-15956: MetadataShell must take the log directory lock when reading [kafka]

Posted by "soarez (via GitHub)" <gi...@apache.org>.
soarez commented on code in PR #14899:
URL: https://github.com/apache/kafka/pull/14899#discussion_r1413167592


##########
shell/src/main/java/org/apache/kafka/shell/MetadataShell.java:
##########
@@ -81,6 +85,55 @@ public MetadataShell build() {
         }
     }
 
+    /**
+     * Return the grandparent directory of a file. This works around Java's quirky API,
+     * which does not honor the UNIX convention of the parent of root being root itself.
+     */
+    static File parentParent(File file) {
+        File parent = file.getParentFile();
+        if (parent == null) {
+            return file;
+        }
+        File parentParent = parent.getParentFile();
+        if (parentParent == null) {
+            return parent;
+        }
+        return parentParent;
+    }

Review Comment:
   ```suggestion
       /**
        * Return the parent directory of a file. This works around Java's quirky API,
        * which does not honor the UNIX convention of the parent of root being root itself.
        */
       static File parent(File file) {
           File parent = file.getParentFile();
           return parent == null ? file : parent;
       }
       
       static File parentParent(File file) {
           return parent(parent(file));
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org