You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2026/01/07 13:28:30 UTC

[PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

sarvekshayr opened a new pull request, #9599:
URL: https://github.com/apache/ozone/pull/9599

   ## What changes were proposed in this pull request?
   - `ContainerToKeyMapping` now handles both FSO and OBS keys as a container may contain both types of keys. 
   
   - Introduced a flag `--onlyFileNames` that allows listing only file names without rebuilding the full FSO tree in case of FSO keys. This significantly reduces processing time when we only need to quickly check whether any keys are affected.
   
   ```
   bash-5.1$ ozone debug om container-key-mapping --containers "1,2" --db /data/metadata/om.db                
   {
     "containers" : {
       "1" : {
         "keys" : [ "vol1/bucket1/OaStVjldJA/ahVKu3nYob/HVYBiF62s9", "vol1/bucket1/OaStVjldJA/ahVKu3nYob/YGpvx930Of", "vol1/bucket1/Y8EAzAebSj/caqb3oJxb2/eyLxOUPwZj", "vol1/bucket1/Y8EAzAebSj/caqb3oJxb2/tyRwN9pnGa", "vol1/bucket1/OaStVjldJA/Po7wJsEmQV", "vol1/bucket1/OaStVjldJA/QTFkKKPla5", "vol1/bucket1/Y8EAzAebSj/a5LvlLLrLs", "vol1/bucket1/Y8EAzAebSj/sDd02ngnXL" ],
         "numOfKeys" : 8
       },
       "2" : {
         "keys" : [ "/vol1/obsbuck/key1" ],
         "numOfKeys" : 1
       }
     }
   }
   bash-5.1$ ozone debug om container-key-mapping --containers "1,2" --db /data/metadata/om.db --onlyFileNames
   {
     "containers" : {
       "1" : {
         "keys" : [ "HVYBiF62s9", "YGpvx930Of", "eyLxOUPwZj", "tyRwN9pnGa", "Po7wJsEmQV", "QTFkKKPla5", "a5LvlLLrLs", "sDd02ngnXL" ],
         "numOfKeys" : 8
       },
       "2" : {
         "keys" : [ "key1" ],
         "numOfKeys" : 1
       }
     }
   }
   
   ```
   
   ## What is the link to the Apache JIRA
   [HDDS-14118](https://issues.apache.org/jira/browse/HDDS-14118)
   
   ## How was this patch tested?
   Added tests in `TestContainerToKeyMapping`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #9599:
URL: https://github.com/apache/ozone/pull/9599


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "priyeshkaratha (via GitHub)" <gi...@apache.org>.
priyeshkaratha commented on code in PR #9599:
URL: https://github.com/apache/ozone/pull/9599#discussion_r2708652227


##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/om/ContainerToKeyMapping.java:
##########
@@ -163,59 +166,102 @@ private void closeDirTreeDB(String dbPath) throws IOException {
     }
   }
 
-  private void retrieve(PrintWriter writer, Set<Long> containerIds) {
-    // Build dir tree
+  private void retrieve(String dbPath, PrintWriter writer, Set<Long> containerIds) {
     Map<Long, Pair<Long, String>> bucketVolMap = new HashMap<>();
-    try {
-      prepareDirIdTree(bucketVolMap);
-    } catch (Exception e) {
-      err().println("Exception occurred reading directory Table, " + e);
-      return;
+    // Build dir tree for FSO keys only if we need full paths
+    if (!onlyFileNames) {
+      try {
+        openDirTreeDB(dbPath);
+        prepareDirIdTree(bucketVolMap);
+      } catch (Exception e) {
+        err().println("Exception occurred reading directory Table, " + e);
+        return;
+      }
     }
 
     // Map to collect keys per container
     Map<Long, List<String>> containerToKeysMap = new HashMap<>();
-    // Track unreferenced keys count per container
+    // Track unreferenced keys count per container (FSO only)
     Map<Long, Long> unreferencedCountMap = new HashMap<>();
     for (Long containerId : containerIds) {
       containerToKeysMap.put(containerId, new ArrayList<>());
       unreferencedCountMap.put(containerId, 0L);
     }
 
-    // Iterate file table and filter for container
-    try (TableIterator<String, ? extends Table.KeyValue<String, OmKeyInfo>> fileIterator = 
+    // Process FSO keys (fileTable)
+    processFSOKeys(containerIds, containerToKeysMap, unreferencedCountMap, bucketVolMap);
+
+    // Process OBS keys (keyTable)
+    processOBSKeys(containerIds, containerToKeysMap);
+
+    jsonOutput(writer, containerToKeysMap, unreferencedCountMap);
+  }

Review Comment:
   nit - please remove unnecessary new lines in the code.  I can see after calling every method a new line is added.



##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/om/ContainerToKeyMapping.java:
##########
@@ -80,22 +79,26 @@ public class ContainerToKeyMapping extends AbstractSubcommand implements Callabl
       description = "Comma separated Container IDs")
   private String containers;
 
+  @CommandLine.Option(names = {"--onlyFileNames"},
+      defaultValue = "false",
+      description = "Only display file names without full path")
+  private boolean onlyFileNames;
+
   private DBStore omDbStore;
   private Table<String, OmVolumeArgs> volumeTable;
   private Table<String, OmBucketInfo> bucketTable;
   private Table<String, OmDirectoryInfo> directoryTable;
   private Table<String, OmKeyInfo> fileTable;
+  private Table<String, OmKeyInfo> keyTable;
   private DBStore dirTreeDbStore;
   private Table<Long, String> dirTreeTable;
   // Cache volume IDs to avoid repeated lookups
   private final Map<String, Long> volumeCache = new HashMap<>();
   private ConfigurationSource conf;
 
-  // TODO: Add support to OBS keys (HDDS-14118)
   @Override
   public Void call() throws Exception {
-    err().println("Note: A container can have both FSO and OBS keys. Currently this tool processes only FSO keys");
-    
+

Review Comment:
   nit - avoid this new line



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "sreejasahithi (via GitHub)" <gi...@apache.org>.
sreejasahithi commented on code in PR #9599:
URL: https://github.com/apache/ozone/pull/9599#discussion_r2703119359


##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/om/ContainerToKeyMapping.java:
##########
@@ -164,7 +168,7 @@ private void closeDirTreeDB(String dbPath) throws IOException {
   }
 
   private void retrieve(PrintWriter writer, Set<Long> containerIds) {
-    // Build dir tree
+    // Build dir tree for FSO keys
     Map<Long, Pair<Long, String>> bucketVolMap = new HashMap<>();
     try {
       prepareDirIdTree(bucketVolMap);

Review Comment:
   The prepareDirIdTree() method is always called which constructs both bucketVolMap and dirTreeTable even when --onlyFileNames=true
   but when --onlyFileNames=true, these are never used since processFSOKeys() directly uses keyInfo.getKeyName().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on PR #9599:
URL: https://github.com/apache/ozone/pull/9599#issuecomment-3752778184

   thanks for the patch @sarvekshayr !
   overall LGTM
   I think the comment in ContainerToKeyMapping.java line 60 should be changed to `Supports both FSO (File System Optimized) and OBS (Object Store) bucket layouts`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on PR #9599:
URL: https://github.com/apache/ozone/pull/9599#issuecomment-3759200264

   > thanks for the patch @sarvekshayr ! overall LGTM I think the comment in ContainerToKeyMapping.java line 60 `only processes FSO layout buckets` should be changed to `Supports both FSO (File System Optimized) and OBS (Object Store) bucket layouts`.
   
   Thanks for pointing it out @rich7420. Fixed it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

Re: [PR] HDDS-14118. Support OBS keys and additional flags in ContainerToKeyMapping tool [ozone]

Posted by "sarvekshayr (via GitHub)" <gi...@apache.org>.
sarvekshayr commented on code in PR #9599:
URL: https://github.com/apache/ozone/pull/9599#discussion_r2703636949


##########
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/debug/om/ContainerToKeyMapping.java:
##########
@@ -164,7 +168,7 @@ private void closeDirTreeDB(String dbPath) throws IOException {
   }
 
   private void retrieve(PrintWriter writer, Set<Long> containerIds) {
-    // Build dir tree
+    // Build dir tree for FSO keys
     Map<Long, Pair<Long, String>> bucketVolMap = new HashMap<>();
     try {
       prepareDirIdTree(bucketVolMap);

Review Comment:
   That's right. Optimised the logic now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org