You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "swamirishi (via GitHub)" <gi...@apache.org> on 2025/08/29 16:41:51 UTC

[PR] HDDS-13612. Track Sst file ranges in sst file metadata [ozone]

swamirishi opened a new pull request, #8981:
URL: https://github.com/apache/ozone/pull/8981

   Change-Id: I822eb168408b65364bc9e009a5818d7f5dfd907f
   
   ## What changes were proposed in this pull request?
   Snapshot metadata should track sst file key ranges. These ranges would be used for snapshot diff. The snapshot meta should also track the previous snapshotId meta to know the version built from which snapshot and the corresponding version.
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-13612
   
   ## How was this patch tested?
   Changed existing unit tests to account for the additional fields


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2331708446


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -602,17 +605,11 @@ private static void deleteKeysFromDelKeyTableInSnapshotScope(
    * @param store AOS or snapshot DB for uncompacted or compacted snapshot respectively.
    * @return a Map of (table, set of SST files corresponding to the table)
    */
-  private static Map<String, Set<String>> getSnapshotSSTFileList(RDBStore store)
+  private static List<LiveFileMetaData> getSnapshotSSTFileList(RDBStore store)
       throws IOException {
-    Map<String, Set<String>> sstFileList = new HashMap<>();
-    List<LiveFileMetaData> liveFileMetaDataList = store.getDb().getLiveFilesMetaData();
-    liveFileMetaDataList.forEach(lfm -> {
-      String cfName = StringUtils.bytes2String(lfm.columnFamilyName());
-      if (COLUMN_FAMILIES_TO_TRACK_IN_SNAPSHOT.contains(cfName)) {
-        sstFileList.computeIfAbsent(cfName, k -> new HashSet<>()).add(lfm.fileName());
-      }
-    });
-    return sstFileList;
+    return store.getDb().getLiveFilesMetaData().stream()

Review Comment:
   I remembered seeing a case where getLiveFilesMetaData() took too long (too many sst files?) and the everything came to a halt. This is used inside OMSnapshotCreateResponse() so if that happens it's going to slow down OM for sure.
   
   But I guess there's no way around it. Perhaps we can add a metric to measure the time spent in snapshot create response handler (do we have that already? not sure) so at least we know what happened.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track Sst file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8981:
URL: https://github.com/apache/ozone/pull/8981#issuecomment-3237663147

   This commit makes significant changes to the way Ozone's snapshot local data tracks and serializes SST (Sorted String Table) file information. Here’s a summary of all changes:
   
   ---
   
   ### 1. **Refactoring SST File Tracking Structure**
   
   - **Old Approach:** SST files (both uncompacted and compacted) were tracked using maps:  
     - `Map<String, Set<String>>` for uncompacted SST file lists  
     - `Map<Integer, Map<String, Set<String>>>` for compacted SST file lists
   
   - **New Approach:**  
     - Introduces a new class `SstFileInfo` to encapsulate SST file metadata: file name, start key, end key, column family.
     - Compacted/uncompacted SST files are now tracked as lists of `SstFileInfo` objects grouped by snapshot version in a new `VersionMeta` class.
     - `VersionMeta` stores:  
       - `previousSnapshotVersion`  
       - `List<SstFileInfo> sstFiles`
     - The maps are replaced by `Map<Integer, VersionMeta> versionSstFileInfos`.
   
   ### 2. **Class and Code Structure Changes**
   
   - **New Class:**  
     - `SstFileInfo` is added, encapsulating SST file details and supporting deep copy, equality, and string representation.
     - `VersionMeta` is added as a static inner class of `OmSnapshotLocalData`, representing a snapshot version's SST file metadata.
   
   - **Existing Classes Modified:**  
     - `CompactionFileInfo` now extends `SstFileInfo` instead of keeping its own fields for SST file metadata.
     - Constructors and methods throughout `OmSnapshotLocalData`, `OmSnapshotLocalDataYaml`, and related test classes are updated to use the new data structures.
   
   ### 3. **YAML Serialization/Deserialization Improvements**
   
   - **Custom YAML tags and representers** for `OmSnapshotLocalDataYaml`, `VersionMeta`, and `SstFileInfo` are introduced for more precise serialization.
   - The code now omits null properties in YAML output.
   - Reading/writing YAML files for snapshot data now uses the new structures and custom serialization logic.
   
   ### 4. **API and Method Changes**
   
   - Methods for getting/setting uncompacted and compacted SST files are replaced with versions that use `VersionMeta` and `SstFileInfo`.
   - Methods like `addUncompactedSSTFileList` and `addCompactedSSTFileList` are replaced with `addVersionSSTFileInfos`.
   
   ### 5. **Test Updates**
   
   - All relevant tests (`TestOmSnapshotLocalDataYaml`, `TestOmSnapshotManager`) are updated to use the new data structures.
   - Mocks for `LiveFileMetaData` now include start/end keys and column family.
   - Assertions now verify the new model (`VersionMeta`, `SstFileInfo`) rather than simple sets/maps.
   
   ### 6. **Constants and Field Additions**
   
   - New string constants added to `OzoneConsts` for the new YAML field names.
   - New fields added for previous snapshot IDs and more detailed SST file tracking.
   
   ### 7. **Code Clean-Up**
   
   - Removes unused imports and code related to the old data structure.
   - Refactors deep copy and equality/hashCode logic to match the new design.
   
   ---
   
   ## **Impact and Motivation**
   
   - **More Structured Metadata:** SST file information is now stored as rich objects instead of simple strings, enabling more precise tracking and manipulation.
   - **Future-Proofing:** The new model is more extensible for future metadata additions.
   - **Serialization Reliability:** Custom YAML serialization ensures the new data structures are reliably persisted and restored.
   - **Test Coverage:** The changes are well covered by updates to existing unit tests.
   
   ---
   
   **In summary:**  
   This commit replaces set/map-based SST file tracking in snapshot local data with a model based on rich metadata objects (`SstFileInfo`, `VersionMeta`), improves YAML serialization/deserialization, and updates all related code and tests to support this more robust and extensible structure.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2336551889


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;

Review Comment:
   I guess we can think about adding some kind of defrag log to a separate file to debug operations similar to the container log file in the datanode



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2336546853


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2323772740


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;
+
+  // Map of version to VersionMeta, using linkedHashMap since the order of the map needs to be deterministic for
+  // checksum computation.
+  private final LinkedHashMap<Integer, VersionMeta> versionSstFileInfos;
 
   public static final Charset CHARSET_ENCODING = StandardCharsets.UTF_8;
   private static final String DUMMY_CHECKSUM = new String(new byte[64], CHARSET_ENCODING);
 
   /**
    * Creates a OmSnapshotLocalData object with default values.
    */
-  public OmSnapshotLocalData(Map<String, Set<String>> uncompactedSSTFileList) {
+  public OmSnapshotLocalData(List<LiveFileMetaData> uncompactedSSTFileList, UUID previousSnapshotId) {

Review Comment:
   yeah the previousSnapshotId in the snapshot chain could be different. Since we are not updating the previous snapshot version value within the same ratis txn it is impossible to achieve atomicity when done via ratis when we have an independent yaml file.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2334235885


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -602,17 +605,11 @@ private static void deleteKeysFromDelKeyTableInSnapshotScope(
    * @param store AOS or snapshot DB for uncompacted or compacted snapshot respectively.
    * @return a Map of (table, set of SST files corresponding to the table)
    */
-  private static Map<String, Set<String>> getSnapshotSSTFileList(RDBStore store)
+  private static List<LiveFileMetaData> getSnapshotSSTFileList(RDBStore store)
       throws IOException {
-    Map<String, Set<String>> sstFileList = new HashMap<>();
-    List<LiveFileMetaData> liveFileMetaDataList = store.getDb().getLiveFilesMetaData();
-    liveFileMetaDataList.forEach(lfm -> {
-      String cfName = StringUtils.bytes2String(lfm.columnFamilyName());
-      if (COLUMN_FAMILIES_TO_TRACK_IN_SNAPSHOT.contains(cfName)) {
-        sstFileList.computeIfAbsent(cfName, k -> new HashSet<>()).add(lfm.fileName());
-      }
-    });
-    return sstFileList;
+    return store.getDb().getLiveFilesMetaData().stream()

Review Comment:
   Eventually we can move this out from the createSnapshot flow and move this to a background service after HDDS-13628



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2334381252


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;

Review Comment:
   nit: add javadoc



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi merged PR #8981:
URL: https://github.com/apache/ozone/pull/8981


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2319490193


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java:
##########
@@ -91,21 +106,77 @@ public static boolean verifyChecksum(OmSnapshotLocalData snapshotData)
     snapshotDataCopy.setChecksum(null);
 
     // Get the YAML representation
-    final Yaml yaml = getYamlForSnapshotLocalData();
+    try (UncheckedAutoCloseableSupplier<Yaml> yaml = snapshotManager.getSnapshotLocalYaml()) {

Review Comment:
   The verifyChecksum method should handle the IOException that can be thrown by getSnapshotLocalYaml() instead of letting it propagate as an unchecked exception. Consider wrapping the IOException in a RuntimeException or adding it to the method signature.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java:
##########
@@ -91,21 +106,77 @@ public static boolean verifyChecksum(OmSnapshotLocalData snapshotData)
     snapshotDataCopy.setChecksum(null);
 
     // Get the YAML representation
-    final Yaml yaml = getYamlForSnapshotLocalData();
+    try (UncheckedAutoCloseableSupplier<Yaml> yaml = snapshotManager.getSnapshotLocalYaml()) {
+      // Compute new checksum
+      snapshotDataCopy.computeAndSetChecksum(yaml.get());
 
-    // Compute new checksum
-    snapshotDataCopy.computeAndSetChecksum(yaml);
+      // Compare the stored and computed checksums
+      String computedChecksum = snapshotDataCopy.getChecksum();
+      boolean isValid = storedChecksum.equals(computedChecksum);
 
-    // Compare the stored and computed checksums
-    String computedChecksum = snapshotDataCopy.getChecksum();
-    boolean isValid = storedChecksum.equals(computedChecksum);
+      if (!isValid) {
+        LOG.warn("Checksum verification failed for snapshot local data. " +
+            "Stored: {}, Computed: {}", storedChecksum, computedChecksum);
+      }
+      return isValid;
+    }
+  }
+
+  /**
+   * Representer class to define which fields need to be stored in yaml file.
+   */
+  private static class OmSnapshotLocalDataRepresenter extends Representer {
+
+    OmSnapshotLocalDataRepresenter(DumperOptions options) {
+      super(options);
+      this.addClassTag(OmSnapshotLocalDataYaml.class, SNAPSHOT_YAML_TAG);
+      this.addClassTag(VersionMeta.class, SNAPSHOT_VERSION_META_TAG);
+      this.addClassTag(SstFileInfo.class, SST_FILE_INFO_TAG);
+      representers.put(SstFileInfo.class, new RepresentSstFileInfo());
+      representers.put(VersionMeta.class, new RepresentVersionMeta());
+      representers.put(UUID.class, data ->
+          new ScalarNode(Tag.STR, data.toString(), null, null, DumperOptions.ScalarStyle.PLAIN));
+    }
+
+    private class RepresentSstFileInfo implements Represent {
+      @Override
+      public Node representData(Object data) {
+        SstFileInfo info = (SstFileInfo) data;
+        Map<String, Object> map = new java.util.LinkedHashMap<>();
+        map.put(OzoneConsts.OM_SST_FILE_INFO_FILE_NAME, info.getFileName());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_START_KEY, info.getStartKey());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_END_KEY, info.getEndKey());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_COL_FAMILY, info.getColumnFamily());
+
+        // Explicitly create a mapping node with the desired tag
+        return representMapping(SST_FILE_INFO_TAG, map, DumperOptions.FlowStyle.BLOCK);
+      }
+    }
+
+    // New inner class for VersionMeta
+    private class RepresentVersionMeta implements Represent {
+      @Override
+      public Node representData(Object data) {
+        VersionMeta meta = (VersionMeta) data;
+        Map<String, Object> map = new java.util.LinkedHashMap<>();
+        map.put(OzoneConsts.OM_SLD_VERSION_META_PREV_SNAP_VERSION, meta.getPreviousSnapshotVersion());
+        map.put(OzoneConsts.OM_SLD_VERSION_META_SST_FILES, meta.getSstFiles());
 
-    if (!isValid) {
-      LOG.warn("Checksum verification failed for snapshot local data. " +
-          "Stored: {}, Computed: {}", storedChecksum, computedChecksum);
+        return representMapping(SNAPSHOT_VERSION_META_TAG, map, DumperOptions.FlowStyle.BLOCK);
+      }
     }

Review Comment:
   The cast to VersionMeta on line 159 is unsafe. Consider adding a type check before casting to prevent ClassCastException if an incorrect object type is passed.



##########
hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/compaction/log/CompactionFileInfo.java:
##########
@@ -117,8 +94,25 @@ public static CompactionFileInfo getFromProtobuf(
 
   @Override
   public String toString() {
-    return String.format("fileName: '%s', startKey: '%s', endKey: '%s'," +
-        " columnFamily: '%s', isPruned: '%b'", fileName, startKey, endKey, columnFamily, pruned);
+    return String.format("%s, isPruned: '%b'", super.toString(), pruned);
+  }
+
+  @Override
+  public SstFileInfo copyObject() {
+    return new CompactionFileInfo(getFileName(), getStartKey(), getEndKey(), getColumnFamily(), pruned);
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (!(o instanceof CompactionFileInfo)) {
+      return false;
+    }
+    return super.equals(o) && pruned == ((CompactionFileInfo)o).pruned;

Review Comment:
   [nitpick] The cast to CompactionFileInfo is unsafe since the type check was performed for CompactionFileInfo. However, consider storing the cast result in a variable for better readability: `CompactionFileInfo other = (CompactionFileInfo) o; return super.equals(o) && pruned == other.pruned;`
   ```suggestion
       CompactionFileInfo other = (CompactionFileInfo) o;
       return super.equals(o) && pruned == other.pruned;
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalDataYaml.java:
##########
@@ -91,21 +106,77 @@ public static boolean verifyChecksum(OmSnapshotLocalData snapshotData)
     snapshotDataCopy.setChecksum(null);
 
     // Get the YAML representation
-    final Yaml yaml = getYamlForSnapshotLocalData();
+    try (UncheckedAutoCloseableSupplier<Yaml> yaml = snapshotManager.getSnapshotLocalYaml()) {
+      // Compute new checksum
+      snapshotDataCopy.computeAndSetChecksum(yaml.get());
 
-    // Compute new checksum
-    snapshotDataCopy.computeAndSetChecksum(yaml);
+      // Compare the stored and computed checksums
+      String computedChecksum = snapshotDataCopy.getChecksum();
+      boolean isValid = storedChecksum.equals(computedChecksum);
 
-    // Compare the stored and computed checksums
-    String computedChecksum = snapshotDataCopy.getChecksum();
-    boolean isValid = storedChecksum.equals(computedChecksum);
+      if (!isValid) {
+        LOG.warn("Checksum verification failed for snapshot local data. " +
+            "Stored: {}, Computed: {}", storedChecksum, computedChecksum);
+      }
+      return isValid;
+    }
+  }
+
+  /**
+   * Representer class to define which fields need to be stored in yaml file.
+   */
+  private static class OmSnapshotLocalDataRepresenter extends Representer {
+
+    OmSnapshotLocalDataRepresenter(DumperOptions options) {
+      super(options);
+      this.addClassTag(OmSnapshotLocalDataYaml.class, SNAPSHOT_YAML_TAG);
+      this.addClassTag(VersionMeta.class, SNAPSHOT_VERSION_META_TAG);
+      this.addClassTag(SstFileInfo.class, SST_FILE_INFO_TAG);
+      representers.put(SstFileInfo.class, new RepresentSstFileInfo());
+      representers.put(VersionMeta.class, new RepresentVersionMeta());
+      representers.put(UUID.class, data ->
+          new ScalarNode(Tag.STR, data.toString(), null, null, DumperOptions.ScalarStyle.PLAIN));
+    }
+
+    private class RepresentSstFileInfo implements Represent {
+      @Override
+      public Node representData(Object data) {
+        SstFileInfo info = (SstFileInfo) data;
+        Map<String, Object> map = new java.util.LinkedHashMap<>();
+        map.put(OzoneConsts.OM_SST_FILE_INFO_FILE_NAME, info.getFileName());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_START_KEY, info.getStartKey());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_END_KEY, info.getEndKey());
+        map.put(OzoneConsts.OM_SST_FILE_INFO_COL_FAMILY, info.getColumnFamily());
+
+        // Explicitly create a mapping node with the desired tag
+        return representMapping(SST_FILE_INFO_TAG, map, DumperOptions.FlowStyle.BLOCK);
+      }
+    }

Review Comment:
   The cast to SstFileInfo on line 144 is unsafe. Consider adding a type check before casting to prevent ClassCastException if an incorrect object type is passed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "smengcl (via GitHub)" <gi...@apache.org>.
smengcl commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2334428210


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;

Review Comment:
   > The snapshot meta should also track the previous snapshotId meta to know the version built from which snapshot and the corresponding version.
   
   It occurred to be that we should have a defrag log somewhere to keep track of thing happening on each OM.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2323730733


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;
+
+  // Map of version to VersionMeta, using linkedHashMap since the order of the map needs to be deterministic for
+  // checksum computation.
+  private final LinkedHashMap<Integer, VersionMeta> versionSstFileInfos;
 
   public static final Charset CHARSET_ENCODING = StandardCharsets.UTF_8;
   private static final String DUMMY_CHECKSUM = new String(new byte[64], CHARSET_ENCODING);
 
   /**
    * Creates a OmSnapshotLocalData object with default values.
    */
-  public OmSnapshotLocalData(Map<String, Set<String>> uncompactedSSTFileList) {
+  public OmSnapshotLocalData(List<LiveFileMetaData> uncompactedSSTFileList, UUID previousSnapshotId) {

Review Comment:
   What is the reason to store previousSnapshotId in the snapshot YAML file when it is already available in the SnapshotInfoTable?
   Do we plan to update the previous snapshot ID whenever there is a change to the snapshot chain i.e., snapshot delete?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track Sst file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8981:
URL: https://github.com/apache/ozone/pull/8981#issuecomment-3237633315

   @copilot summarise the changes in the patch and review the PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8981:
URL: https://github.com/apache/ozone/pull/8981#issuecomment-3275361754

   thank you for reviewing the patch @jojochuang @smengcl @SaketaChalamchala 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "SaketaChalamchala (via GitHub)" <gi...@apache.org>.
SaketaChalamchala commented on code in PR #8981:
URL: https://github.com/apache/ozone/pull/8981#discussion_r2323730733


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotLocalData.java:
##########
@@ -45,32 +51,33 @@ public abstract class OmSnapshotLocalData {
   // Whether SST is filtered
   private boolean isSSTFiltered;
 
-  // Map of Table to uncompacted SST file list on snapshot create
-  private Map<String, Set<String>> uncompactedSSTFileList;
-
   // Time of last compaction, in epoch milliseconds
   private long lastCompactionTime;
 
   // Whether the snapshot needs compaction
   private boolean needsCompaction;
 
-  // Map of version to compacted SST file list
-  // Map<version, Map<Table, sstFileList>>
-  private Map<Integer, Map<String, Set<String>>> compactedSSTFileList;
+  private UUID previousSnapshotId;
+
+  // Map of version to VersionMeta, using linkedHashMap since the order of the map needs to be deterministic for
+  // checksum computation.
+  private final LinkedHashMap<Integer, VersionMeta> versionSstFileInfos;
 
   public static final Charset CHARSET_ENCODING = StandardCharsets.UTF_8;
   private static final String DUMMY_CHECKSUM = new String(new byte[64], CHARSET_ENCODING);
 
   /**
    * Creates a OmSnapshotLocalData object with default values.
    */
-  public OmSnapshotLocalData(Map<String, Set<String>> uncompactedSSTFileList) {
+  public OmSnapshotLocalData(List<LiveFileMetaData> uncompactedSSTFileList, UUID previousSnapshotId) {

Review Comment:
   What is the reason to store previousSnapshotId in the snapshot YAML file when it is already available in the SnapshotInfoTable?
   Do we plan to update the previous snapshot ID in the YAML whenever there is a change to the snapshot chain as well i.e., snapshot delete?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-13612. Track SST file ranges in sst file metadata [ozone]

Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #8981:
URL: https://github.com/apache/ozone/pull/8981#issuecomment-3261943045

   @smengcl @SaketaChalamchala @jojochuang Can you take a look at this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org