[GitHub] [ozone] smengcl opened a new pull request, #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl opened a new pull request, #4567:
URL: https://github.com/apache/ozone/pull/4567

   ## What changes were proposed in this pull request?
   
   This is approach 1 of 2 that could fix the issue that bothers SnapDiff, where the current `LoadingCache` behaves like a simple LRU cache. We had no control over when an `OmSnapshot` instance can be evicted and closed, which can cause the the snapshot DB instance to be **closed prematurely** while SnapDiff is still running in the background, crashing the OM.
   
   Opening this as a draft for suggestions. @GeorgeJahad @hemantk-12 could also take a look at this. I'd like to give some opinions and feedbacks on the overall approach.
   
   The core of this PR is `SnapshotCache` class and `RefCount` interface. One can start from there. `SnapshotCache` is supposed to be thread-safe.
   
   With this approach, every single time an `OmSnapshot` instance is retrieved using `snapshotCache.get()`, a corresponding `snapshotCache.release()` has to be called at the end of its usage, otherwise it causes leakage because the reference count is not decremented. I have identified all existing usages of `snapshotCache.get()`, wrapped in another helper method or not, on latest master branch and added the logic (or TODO) there on how they should be properly handled.
   
   One can very, very easily forget to release the handle (as we have seen with all the RDB handle leaks Duong has been painstakingly dealing with over the past few months), so I don't think this approach is the cleanest. However, putting the decrement ref count logic in `OmSnapshot#close` (so that we can wrap `OmSnapshot` inside a try-with-resources) would also break the encapsulation thus considered hacky by myself.
   
   I put sanity checks everywhere just to be defensive.
   
   In the meantime, I will explore if [`CacheBuilder#weakValues()`](https://www.javadoc.io/doc/com.google.guava/guava/latest/com/google/common/cache/CacheBuilder.html#weakValues()) would be a viable alternative (potential approach 2), where we could probably rely on JVM itself to do the "reference counting" on the instances.
   
   ## What is the link to the Apache JIRA
   
   https://issues.apache.org/jira/browse/HDDS-7935
   
   ## How was this patch tested?
   
   - [ ] Unit test to be finished.
   - [ ] Tests that covers existing snapshot cache use-cases, that is all snapshot operations that read or write snapshot DB, should all pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234532031


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -246,7 +248,7 @@ public class OmMetadataManagerImpl implements OMMetadataManager,
 
   private DBStore store;
 
-  private final IOzoneManagerLock lock;
+  private IOzoneManagerLock lock;

Review Comment:
   why not final?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225195161


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestSnapshotDeletingService.java:
##########
@@ -471,4 +471,3 @@ private boolean assertTableRowCount(int expectedCount,
     return count == expectedCount;
   }
 }
-

Review Comment:
   Ah, I see. I think you remove an extra line and got confused. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1244568291


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {
+      if (v == null) {
+        throw new IllegalArgumentException(
+            "Key '" + key + "' does not exist in cache");
       }
-    }
+
+      if (v.decrementRefCount() == 0L) {
+        synchronized (pendingEvictionList) {
+          // Eligible to be closed, add it to the list.
+          pendingEvictionList.add(v);
+          // The cache size might have already exceed the soft limit
+          // Thus triggering cleanup() to check and evict if applicable
+          cleanup();
+        }
+      }
+
+      return v;

Review Comment:
   Good point. Replaced this line with `return dbMap.get(k);`
   
   Might look hacky, but it works. Tested with a snippet:
   
   ```java
   import java.util.concurrent.ConcurrentHashMap;
   
   class Untitled {
     public static void main(String[] args) {
       ConcurrentHashMap map = new ConcurrentHashMap<String, String>();
       String k1 = "k1";
       map.put(k1, "val1");
       map.compute(k1, (k, v) -> {
         System.out.format("k=%s, v=%s\n", k, v);
         map.remove(k);
         return map.get(k);
       });
       System.out.format("k=%s, v=%s\n", k1, map.get(k1));
     }
   }
   ```
   
   Output:
   ```
   k=k1, v=val1
   k=k1, v=null
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1509225002

   > I also think some of the snapshot user code should be modified to support try-with-resources. For example, this code:
   > 
   > https://github.com/apache/ozone/blob/ddfc4b738d6d48ef37f3af41a7f743247933e30a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java#L83-L91
   > 
   > does the checkForSnapshot() call and then passes a reference to the omSnapshot into the double buffer thread.
   > 
   > I think those sorts of calls should be modified just to pass the snapshot key into the double buffer thread and have it get the snapshot from the cache, (with try-with-resources.)
   > 
   > The way it is implemented now, the reference is passed between two threads, and try-with-resources requires it all happen in the same thread.
   
   Back when @aswinshakil was implementing this in #4486 I [asked the same question](https://github.com/apache/ozone/pull/4486#discussion_r1151192924) of why didn't we just pass the snapshot table key to the `Response` and get the instance there. The answer was also encapsulation. `Response` only gets the `OMMetadataManager` instance (active or snapshot DB) it is operating on. Though I think passing in the extra `omSnapshotManager` alone is fine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239067802


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    synchronized (pendingEvictionList) {
+      // Remove instance from clean up list when it exists.
+      pendingEvictionList.remove(rcOmSnapshot);
+      // Check if any entries can be cleaned up.
+      // At this point, cache size might temporarily exceed cacheSizeLimit
+      // even if there are entries that can be evicted, which is fine since it
+      // is a soft limit.
+      cleanup();
+    }
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      synchronized (pendingEvictionList) {

Review Comment:
   I believe you mean "it will **decrement** the count"?
   
   Should be fixed by the [change](https://github.com/apache/ozone/pull/4567#discussion_r1239066138) mentioned above because ref count is now increased as soon as it gets the RC instance from `dbMap`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1222312569


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   No worries. :D
   
   The native approach would be to O(1) scan the `dbMap` every time `cleanup()` is called, but that obviously wouldn't be ideal.
   
   Only if there is a `ConcurrentTreeMap` that can sort based on **values**.
   
   Or I could bake a doubly linked list inside `ReferenceCounted` so each `ReferenceCounted<>` instance becomes a node..



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   No worries. :D
   
   The naive approach would be to O(1) scan the `dbMap` every time `cleanup()` is called, but that obviously wouldn't be ideal.
   
   Only if there is a `ConcurrentTreeMap` that can sort based on **values**.
   
   Or I could bake a doubly linked list inside `ReferenceCounted` so each `ReferenceCounted<>` instance becomes a node..



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234532089


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -293,14 +292,26 @@ public ReentrantReadWriteLock getTableLock(String tableName) {
   // non-ratis OM clusters will be binary 01 (= decimal 1)  and for ratis
   // enabled OM cluster will be binary 10 (= decimal 2). This epoch is added
   // to ensure uniqueness of objectIDs.
-  private final long omEpoch;
+  private long omEpoch;

Review Comment:
   why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225176880


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);
+      cleanup();
+    }
+  }
+
+  /**
+   * Alternatively, can release with OmSnapshot instance directly.
+   * @param omSnapshot OmSnapshot
+   */
+  public void release(OmSnapshot omSnapshot) {
+    final String key = omSnapshot.getSnapshotTableKey();
+    release(key);
+  }
+
+  /**
+   * Callback method used to enqueue or dequeue ReferenceCounted from
+   * pendingEvictionList.
+   * @param referenceCounted ReferenceCounted object
+   */
+  public void callback(ReferenceCounted referenceCounted) {

Review Comment:
   I wanted to. But can't seem to do this elegantly.
   
   Adding `<IOmMetadataReader>` makes it uncallable from `ReferenceCounted` because the latter template takes `<T>`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1584968462

   I see you already implemented your approach and I think it is fine.  I just wanted to show you what I meant:
   
   
   
   Create an interface like this:
   https://github.com/GeorgeJahad/ozone/compare/siyaosOriginal..addEvictable#diff-dcbfe3b8f20b10894bacf32579a63703629eaf1d38543c696ca786efdbe4cbe3R1-R7
   
   use it like this:
   https://github.com/GeorgeJahad/ozone/compare/siyaosOriginal..addEvictable#diff-c624eeb3f5cb3e828f7218adb6cc7aca1f0108ff1a30c8a2a456f2d0b027880bR120-R122
   
   and implement it like this:
   https://github.com/GeorgeJahad/ozone/compare/siyaosOriginal..addEvictable#diff-d5b0b24274c5d19e876dc2ffeb744d4965fa1a23ce727ac09b965f665573c18aR328-R344
   
   The rest of the necessary changes here:
   https://github.com/GeorgeJahad/ozone/compare/siyaosOriginal..addEvictable
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231749293


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231747963


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?

Review Comment:
   nice. thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1509137434

   I tend to prefer approach 1.
   
   The guava docs has specific warnings about using weak/soft values: https://guava.dev/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html#weakValues()
   
   "Weak values will be garbage collected once they are weakly reachable. This makes them a poor candidate for caching; consider softValues() instead. "
   
   "Warning: in most circumstances it is better to set a per-cache maximum size instead of using soft references. You should only use this method if you are well familiar with the practical consequences of soft references. "
   
   So I am hesitant about approach 2.
   
   For me the main problem with approach 1 is that the implementation doesn't yet support try-with-resources.
   
   Your concern about encapsulation is valid: As you say, "putting the decrement ref count logic in OmSnapshot#close (so that we can wrap OmSnapshot inside a try-with-resources) would also break the encapsulation thus considered hacky by myself."
   
   Would it be better, if instead of the refCount interface, we had a cacheEntry wrapper class, that manages the reference counting and also implements the close?  The cache entry constructor would take an omSnapshot as a paramater, and would have a getter method to all the enduser to access the actual omSnapshot methods?
   
   So users would always do a try-with-resources{} to get a cache entry and then unwrap that to get the omSnapshot?  Or maybe we could make the cache entry implement the IOMetadataReader interface as well, so there would usually be no unwrapping needed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232889107


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   ok, but now I'm not quite sure how the underlying cache is stored.  Does it have it's own eviction policy?
   
   Also, I find it confusing to have the removalListener implemented but not having it be used anywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1244618933


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {
+      if (v == null) {
+        throw new IllegalArgumentException(
+            "Key '" + key + "' does not exist in cache");
       }
-    }
+
+      if (v.decrementRefCount() == 0L) {
+        synchronized (pendingEvictionList) {
+          // Eligible to be closed, add it to the list.
+          pendingEvictionList.add(v);
+          // The cache size might have already exceed the soft limit
+          // Thus triggering cleanup() to check and evict if applicable
+          cleanup();
+        }
+      }
+
+      return v;

Review Comment:
   nvm. turns out it is a bad idea to update other mappings inside a `map.compute()` anyways according to its javadoc. In this case it corrupts the map size count, and it caught by my UT:
   
   https://github.com/apache/ozone/actions/runs/5396424029/jobs/9800041580?pr=4567#step:6:2594
   
   Repro snippet:
   
   ```java
   import java.util.concurrent.ConcurrentHashMap;
   
   class Untitled {
     public static void main(String[] args) {
       ConcurrentHashMap map = new ConcurrentHashMap<String, String>();
       String k1 = "k1";
       map.put(k1, "v1");
       map.put("k2", "v2");
       System.out.println("- Map dump 1");
       map.forEach((k, v) -> {
         System.out.format("k=%s, v=%s\n", k, v, map.size());
       });
       System.out.println();
       
       System.out.println("- compute()");
       map.compute(k1, (k, v) -> {
         System.out.format("k=%s, v=%s, size=%d\n", k, v, map.size());
         map.remove(k);
         System.out.format("k=%s, v=%s, size=%d\n", k, v, map.size());
         return map.get(k);
       });
       System.out.println();
       // Map size becomes incorrect at this point
       System.out.format("k=%s, v=%s, size=%d\n", k1, map.get(k1), map.size());
       
       System.out.println("- Map dump 2");
       map.forEach((k, v) -> {
         System.out.format("k=%s, v=%s\n", k, v, map.size());
       });
     }
   }
   ```
   
   Output:
   ```
   - Map dump 1
   k=k1, v=v1
   k=k2, v=v2
   
   - compute()
   k=k1, v=v1, size=2
   k=k1, v=v1, size=1
   
   k=k1, v=null, size=0
   - Map dump 2
   k=k2, v=v2
   ```
   
   You can see map's `size=0` while it obviously still has `k2` entry in it.
   
   I have moved `cleanup()` out of `compute()` and added additional syncing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239108575


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotMoveDeletedKeysResponse.java:
##########
@@ -78,42 +86,74 @@ public OMSnapshotMoveDeletedKeysResponse(@Nonnull OMResponse omResponse) {
   protected void addToDBBatch(OMMetadataManager omMetadataManager,
       BatchOperation batchOperation) throws IOException {
 
-    if (nextSnapshot != null) {
-      RDBStore nextSnapshotStore =
-          (RDBStore) nextSnapshot.getMetadataManager().getStore();
-      // Init Batch Operation for snapshot db.
-      try (BatchOperation writeBatch = nextSnapshotStore.initBatchOperation()) {
-        processKeys(writeBatch, nextSnapshot.getMetadataManager());
-        processDirs(writeBatch, nextSnapshot.getMetadataManager());
-        nextSnapshotStore.commitBatchOperation(writeBatch);
-        nextSnapshotStore.getDb().flushWal(true);
-        nextSnapshotStore.getDb().flush();
+    // Note: a trick to get
+    // To do this properly, refactor OzoneManagerDoubleBuffer#addToBatch and
+    // add OmSnapshotManager as a parameter.

Review Comment:
   If I hadn't changed `OmMetadataManagerImpl` constructor I might have taken this "lazy" approach. IMO that doesn't seem to be a memory-efficient way to pass along something that could be retrieved from the active OM. It is occupying at least 8 bytes (64-bit pointer plus any JVM overhead) for each such request that are not flushed in the double buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1612144025

   That looks like it fixes the issues I found.  Thanks!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1509213712

   Thanks @GeorgeJahad for the comment.
   
   > I tend to prefer approach 1.
   > 
   > The guava doc has specific warnings about using weak/soft values: [guava.dev/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html#weakValues()](https://guava.dev/releases/19.0/api/docs/com/google/common/cache/CacheBuilder.html#weakValues())
   > 
   > "Weak values will be garbage collected once they are weakly reachable. This makes them a poor candidate for caching; consider softValues() instead. "
   > 
   > "Warning: in most circumstances it is better to set a per-cache maximum size instead of using soft references. You should only use this method if you are well familiar with the practical consequences of soft references. "
   > 
   > So I am hesitant about approach 2.
   
   Yes I read the doc before I went through the implementation. Though I should have used `softValues()` in approach 2. Also just found an unbounded cache usage in Hive code base here:
   
   https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L76-L80
   
   > For me the main problem with approach 1 is that the implementation doesn't yet support try-with-resources.
   > 
   > Your concern about encapsulation is valid: As you say, "putting the decrement ref count logic in OmSnapshot#close (so that we can wrap OmSnapshot inside a try-with-resources) would also break the encapsulation thus considered hacky by myself."
   > 
   > Would it be better, if instead of the refCount interface, we had a cacheEntry wrapper class, that manages the reference counting and also implements the close? The cache entry constructor would take an omSnapshot as a paramater, and would have a getter method to all the enduser to access the actual omSnapshot methods?
   > 
   > So users would always do a try-with-resources{} to get a cache entry and then unwrap that to get the omSnapshot? Or maybe we could make the cache entry implement the IOMetadataReader interface as well, so there would usually be no unwrapping needed.
   
   That could work. I had the same idea briefly but I am not too fond of the extra layer either. However, this would preserve the encapsulation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1515136255

   @hemantk-12 
   
   > I don't see cacheSize is used in eviction policy if cache is full. Yes, it doesn't proactively claims some entries but only if they are not used by anyone. If it need bounded, cache it doesn't solve that problem.
   
   It does seem to be implemented here, or am I misunderstanding your point?
   https://github.com/apache/ozone/blob/ddfc4b738d6d48ef37f3af41a7f743247933e30a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java#L167-L174
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1515135518

   @smengcl I have no experience with "softValues()" but my understanding is that it won't do any reclamation until the heap is completely full.  Is that correct?
   
   > In a degenerate case where OM JVM heap usage is pushed to the absolute limit,
   
   My fear is that this is not a degenerate case but will rather be common once enough snapshot operations have been performed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231740949


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   @GeorgeJahad 
   
   That shouldn't be a problem with the current implementation, we are calling `close()` on a different level. Even if `ReferenceCounted#close` is called when the wrapped `IOmMetadataReader` is an `OzoneManager` instance, it is only decrementing the reference count (if RC is enabled on active DB).
   
   Because active DB will never be in `SnapshotCache`, it is nevered queued in `pendingEvictionList`. Thus, `OzoneManager#close` will never be called in the case of RC.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1230268391


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }

Review Comment:
   also have to synchronize refcount checks with eviction list mods, see https://github.com/apache/ozone/pull/4567/files/98eff381cb857d0ac5b58a8b21e050becf638c37#r1230267684



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234553698


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotMoveDeletedKeysResponse.java:
##########
@@ -78,42 +86,74 @@ public OMSnapshotMoveDeletedKeysResponse(@Nonnull OMResponse omResponse) {
   protected void addToDBBatch(OMMetadataManager omMetadataManager,
       BatchOperation batchOperation) throws IOException {
 
-    if (nextSnapshot != null) {
-      RDBStore nextSnapshotStore =
-          (RDBStore) nextSnapshot.getMetadataManager().getStore();
-      // Init Batch Operation for snapshot db.
-      try (BatchOperation writeBatch = nextSnapshotStore.initBatchOperation()) {
-        processKeys(writeBatch, nextSnapshot.getMetadataManager());
-        processDirs(writeBatch, nextSnapshot.getMetadataManager());
-        nextSnapshotStore.commitBatchOperation(writeBatch);
-        nextSnapshotStore.getDb().flushWal(true);
-        nextSnapshotStore.getDb().flush();
+    // Note: a trick to get
+    // To do this properly, refactor OzoneManagerDoubleBuffer#addToBatch and
+    // add OmSnapshotManager as a parameter.

Review Comment:
   Couldn't the snapshotManager just be added as a parameter to the OMSnapshotMoveDeletedKeysResponse constructor, similiar to how it is done for the secretManager here: https://github.com/apache/ozone/blob/016a433a06506714dcfea5ccb806f758770135e1/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/s3/security/OMSetSecretResponse.java#L51-L60



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240021963


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   Sorry if I'm being dense, but I'm still not following.  What is the actual data structure currently storing the cache?  Is it a com.google.common.cache.CacheLoader?  What kind of cache is that?  Does it have an eviction policy?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234532089


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -293,14 +292,26 @@ public ReentrantReadWriteLock getTableLock(String tableName) {
   // non-ratis OM clusters will be binary 01 (= decimal 1)  and for ratis
   // enabled OM cluster will be binary 10 (= decimal 2). This epoch is added
   // to ensure uniqueness of objectIDs.
-  private final long omEpoch;
+  private long omEpoch;

Review Comment:
   why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225170674


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   > Is it possible that release can be called twice on the same key?
   Only when `get()` is also called twice (or more times). Otherwise it is a programmatial error.
   
   One `release()` (or `close()`) has to paired with one previous `get()`. Calling `release()` twice with only one previous `get()` would result in referenced count being decremented to an undefined `-1`.
   
   Just like one `unlock()` would pair to one previous `lock()` for `ReentrantLock`.
   
   Also, the entry is not evicted as soon as `release()` is called in this case. It could be marked as "eligible to be evicted" by adding it to the `pendingEvictionList`



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   > Is it possible that release can be called twice on the same key?
   
   Only when `get()` is also called twice (or more times). Otherwise it is a programmatial error.
   
   One `release()` (or `close()`) has to paired with one previous `get()`. Calling `release()` twice with only one previous `get()` would result in referenced count being decremented to an undefined `-1`.
   
   Just like one `unlock()` would pair to one previous `lock()` for `ReentrantLock`.
   
   Also, the entry is not evicted as soon as `release()` is called in this case. It could be marked as "eligible to be evicted" by adding it to the `pendingEvictionList`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl merged pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl merged PR #4567:
URL: https://github.com/apache/ozone/pull/4567


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232844063


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   Hi @GeorgeJahad , the plan was to do `softValues()` first (because that was much easier and quicker to impl), then do custom cache impl to entirely replace the `LoadingCache` (as the title suggests).
   
   We recently are suspecting a small issue with `softValues()`, where seemingly too many instances are open at a time and each are holding quite a few files open. Thus we filed #4884 as a mitigation.
   
   It is plausible to make the interface pluggable. But with the only advantage of rolling back to `LoadingCache` impl when custom cache impl goes wrong. Would you like to file a new jira for this? (requires adding a new OM config, adding a new interface, tweaking `RC#close` / `decrementRefCount`, etc.)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231714733


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);

Review Comment:
   if `threadMap` is null, `refCount` must have been null as well.
   
   adding the extra check to make intellij happy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231730355


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }
+
+      return newVal;
+    });
+
+    return refCount.get();
+  }
+
+  public long decrementRefCount() {
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    Preconditions.checkState(threadMap.containsKey(tid),
+        "Current thread have not holden reference before");
+
+    Preconditions.checkNotNull(threadMap.get(tid), "This thread " + tid +
+        " has not incremented the reference count before.");

Review Comment:
   1. Yeah the checks are there just to be paranoid.
   2. Makes sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234543967


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMDirectoriesPurgeRequestWithFSO.java:
##########
@@ -148,7 +151,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
         getOmRequest());
     OMClientResponse omClientResponse = new OMDirectoriesPurgeResponseWithFSO(
         omResponse.build(), purgeRequests, ozoneManager.isRatisEnabled(),
-            getBucketLayout(), volBucketInfoMap, omFromSnapshot);
+            getBucketLayout(), volBucketInfoMap, rcOmFromSnapshot);

Review Comment:
   In OMKeyPurgeRequest.java, you changed the parameter from snapshot to snapshotInfo.  Do we want to do the same thing here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1515174384

   > 
   
   No, it doesn't because it is called only when `release()` is called.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1222185667


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   `ReferenceCounted<>` are inside a map in `SnapshotCache`. So that would require me to pass the reference to `SnapshotCache` every time I create a new `ReferenceCounted<>` right?
   
   Is there a better way?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1220669021


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   Ah I see, `ReferenceCounted::close` does not directly trigger `SnapshotCache::release` right now.
   
   I would need to incorporate the `pendingEvictionList` logic in there somehow. So that `cleanup()` would act correctly upon cache eviction.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1224624077


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?

Review Comment:
   If it has to be Concurrent `LinkedHashSet`, I don't think there any library which can be used directly. You can do something similar to https://stackoverflow.com/a/1391955 and use `Collections.synchronizedSet()`.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);

Review Comment:
   null check on `threadMap` because it can be null. Or do an assert.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   Is it possible that release can be called twice on the same key? If yes, then this condition check would fail. In that scenario, I don't think it is good idea to have this condition check. 



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically

Review Comment:
   nit: in my opinion these type of comments are not needed. `putIfAbsent` (method name) says itself that it will put if key is absent and `threadMap` is concurrent `HashMap` so operation would be atomic.
   In short they are not adding any value to me.
   
   Same for line 83.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);
+      cleanup();
+    }
+  }
+
+  /**
+   * Alternatively, can release with OmSnapshot instance directly.
+   * @param omSnapshot OmSnapshot
+   */
+  public void release(OmSnapshot omSnapshot) {
+    final String key = omSnapshot.getSnapshotTableKey();
+    release(key);
+  }
+
+  /**
+   * Callback method used to enqueue or dequeue ReferenceCounted from
+   * pendingEvictionList.
+   * @param referenceCounted ReferenceCounted object
+   */
+  public void callback(ReferenceCounted referenceCounted) {
+    if (referenceCounted.getTotalRefCount() == 0L) {
+      // Reference count reaches zero, add to pendingEvictionList
+      Preconditions.checkState(!pendingEvictionList.contains(referenceCounted),
+          "SnapshotCache is inconsistent. Entry should not be in the "
+              + "pendingEvictionList when ref count just reached zero.");
+      pendingEvictionList.add(referenceCounted);
+    } else if (referenceCounted.getTotalRefCount() == 1L) {
+      pendingEvictionList.remove(referenceCounted);
+    }
+  }
+
+  /**
+   * If cache size exceeds soft limit, attempt to clean up and close the
+   * instances that has zero reference count.
+   * TODO: [SNAPSHOT] Add new ozone debug CLI command to trigger this directly.
+   */
+  private void cleanup() {
+    long numEntriesToEvict = (long) dbMap.size() - cacheSizeLimit;
+    while (pendingEvictionList.size() > 0 && numEntriesToEvict > 0L) {
+      // Get the first instance in the clean up list
+      ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+          pendingEvictionList.iterator().next();
+      OmSnapshot omSnapshot = (OmSnapshot) rcOmSnapshot.get();
+      LOG.debug("Evicting OmSnapshot instance {} with table key {}",
+          rcOmSnapshot, omSnapshot.getSnapshotTableKey());
+      // Sanity check
+      Preconditions.checkState(rcOmSnapshot.getTotalRefCount() == 0L,
+          "Illegal state: OmSnapshot reference count non-zero ("
+              + rcOmSnapshot.getTotalRefCount() + ") but shows up in the "
+              + "clean up list");
+
+      final String key = omSnapshot.getSnapshotTableKey();
+      final ReferenceCounted<IOmMetadataReader> result = dbMap.remove(key);
+      // Sanity check
+      Preconditions.checkState(rcOmSnapshot == result,
+          "Cache map entry removal failure. The cache is in an inconsistent "
+              + "state. Expected OmSnapshot instance: " + rcOmSnapshot
+              + ", actual: " + result);
+
+      pendingEvictionList.remove(result);
+
+      // Close the instance, which also closes its DB handle.
+      try {
+        ((OmSnapshot) rcOmSnapshot.get()).close();
+      } catch (IOException ex) {
+        throw new IllegalStateException("Error while closing snapshot DB", ex);
+      }
+
+      --numEntriesToEvict;
+    }
+
+    // Print warning message if actual cache size is exceeding the soft limit
+    // even after the cleanup procedure above.
+    if ((long) dbMap.size() > cacheSizeLimit) {
+      LOG.warn("Current snapshot cache size ({}) is exceeding configured "
+          + "soft-limit ({}) after possible evictions.",
+          dbMap.size(), cacheSizeLimit);
+
+      Preconditions.checkState(pendingEvictionList.size() == 0);

Review Comment:
   I'm skeptical about it and would prefer to add it in if block because there is no synchronization between this and `pendingEvictionList.add(referenceCounted);` at line # 249.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.

Review Comment:
   nit:
   ```suggestion
       // when called from SDT (and some) if the snapshot is not active anymore.
   ```



##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestSnapshotDeletingService.java:
##########
@@ -471,4 +471,3 @@ private boolean assertTableRowCount(int expectedCount,
     return count == expectedCount;
   }
 }
-

Review Comment:
   Why? :P
   
   I think every file should have an empty line in the end of the file.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,

Review Comment:
   I am not aligned by providing the flexibility to `disableCounter`. If someone doesn't need reference count, then don't wrap object with `ReferenceCounted`. I see it is used for active DB to be in consistency but you can still keep the count. It is not harmful, IMO. Providing `disableCounter` is just making it over complicated. Also defeats the whole purpose of reference count.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }
+
+      return newVal;
+    });
+
+    return refCount.get();
+  }
+
+  public long decrementRefCount() {
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    Preconditions.checkState(threadMap.containsKey(tid),
+        "Current thread have not holden reference before");
+
+    Preconditions.checkNotNull(threadMap.get(tid), "This thread " + tid +
+        " has not incremented the reference count before.");

Review Comment:
   1. Are these precondition checks to make sure that `decrementRefCount` and `incrementRefCount` are not getting called two or more times for the same object? And catch any bug?
   2. These is no difference in`Preconditions.checkState(threadMap.containsKey(tid), ...)` and `Preconditions.checkNotNull(threadMap.get(tid),..)`. Because if `threadMap` contains tid and value is 0L (unboxed), its boxed value would be 0L(boxed) not null. Second condition should be `Preconditions.checkState(threadMap.get(tid) > 0L, ..)`



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;

Review Comment:
   Is it has to be `SnapshotCache`? Should it type T or P? `ReferenceCounted` is not generic anymore and getting coupled with Snapshot. I'm fine if you make `ReferenceCounted` for snapshot only but then it should not be T type.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;

Review Comment:
   What do we want to achieve with this `threadMap`? I am not seeing any use of it as of now.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&

Review Comment:
   nit: to fail fast.
   ```suggestion
       if (!skipActiveCheck &&
           !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE))
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }

Review Comment:
   This should be move out of `computeIfPresent`. I believe you did it this way to synchronize global and thread level  update. It would be cleaner if you use synchronize block on `refCount`.
   
   ```
       synchronized (refCount) {
         threadMap.computeIfPresent(tid, (k, v) -> {
           long newVal = v + 1;
           Preconditions.checkState(newVal > 0L,
               "Thread reference count overflown");
           return newVal;
         });
   
         long newValTotal = refCount.incrementAndGet();
         Preconditions.checkState(newValTotal > 0L,
             "Total reference count overflown");
   
         if (refCount.get() == 1L) {
           // ref count increased to one (from zero), remove from
           // pendingEvictionList if added
           parentSnapshotCache.callback(this);
         }
       }
   ```
   
   same for `decrementRefCount`.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);
+      cleanup();
+    }
+  }
+
+  /**
+   * Alternatively, can release with OmSnapshot instance directly.
+   * @param omSnapshot OmSnapshot
+   */
+  public void release(OmSnapshot omSnapshot) {
+    final String key = omSnapshot.getSnapshotTableKey();
+    release(key);
+  }
+
+  /**
+   * Callback method used to enqueue or dequeue ReferenceCounted from
+   * pendingEvictionList.
+   * @param referenceCounted ReferenceCounted object
+   */
+  public void callback(ReferenceCounted referenceCounted) {
+    if (referenceCounted.getTotalRefCount() == 0L) {
+      // Reference count reaches zero, add to pendingEvictionList
+      Preconditions.checkState(!pendingEvictionList.contains(referenceCounted),
+          "SnapshotCache is inconsistent. Entry should not be in the "
+              + "pendingEvictionList when ref count just reached zero.");
+      pendingEvictionList.add(referenceCounted);
+    } else if (referenceCounted.getTotalRefCount() == 1L) {
+      pendingEvictionList.remove(referenceCounted);
+    }
+  }
+
+  /**
+   * If cache size exceeds soft limit, attempt to clean up and close the
+   * instances that has zero reference count.
+   * TODO: [SNAPSHOT] Add new ozone debug CLI command to trigger this directly.
+   */
+  private void cleanup() {
+    long numEntriesToEvict = (long) dbMap.size() - cacheSizeLimit;
+    while (pendingEvictionList.size() > 0 && numEntriesToEvict > 0L) {

Review Comment:
   nit: fail fast
   ```suggestion
       while (numEntriesToEvict > 0L && pendingEvictionList.size() > 0) { 
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);
+      cleanup();
+    }
+  }
+
+  /**
+   * Alternatively, can release with OmSnapshot instance directly.
+   * @param omSnapshot OmSnapshot
+   */
+  public void release(OmSnapshot omSnapshot) {
+    final String key = omSnapshot.getSnapshotTableKey();
+    release(key);
+  }
+
+  /**
+   * Callback method used to enqueue or dequeue ReferenceCounted from
+   * pendingEvictionList.
+   * @param referenceCounted ReferenceCounted object
+   */
+  public void callback(ReferenceCounted referenceCounted) {

Review Comment:
   Make it typed.
   ```suggestion
     public void callback(ReferenceCounted<IOmMetadataReader> referenceCounted)
   ```



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()

Review Comment:
   I'm guessing you meant the `pendingEvictionList.remove(result);` in cleanup().



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239108575


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotMoveDeletedKeysResponse.java:
##########
@@ -78,42 +86,74 @@ public OMSnapshotMoveDeletedKeysResponse(@Nonnull OMResponse omResponse) {
   protected void addToDBBatch(OMMetadataManager omMetadataManager,
       BatchOperation batchOperation) throws IOException {
 
-    if (nextSnapshot != null) {
-      RDBStore nextSnapshotStore =
-          (RDBStore) nextSnapshot.getMetadataManager().getStore();
-      // Init Batch Operation for snapshot db.
-      try (BatchOperation writeBatch = nextSnapshotStore.initBatchOperation()) {
-        processKeys(writeBatch, nextSnapshot.getMetadataManager());
-        processDirs(writeBatch, nextSnapshot.getMetadataManager());
-        nextSnapshotStore.commitBatchOperation(writeBatch);
-        nextSnapshotStore.getDb().flushWal(true);
-        nextSnapshotStore.getDb().flush();
+    // Note: a trick to get
+    // To do this properly, refactor OzoneManagerDoubleBuffer#addToBatch and
+    // add OmSnapshotManager as a parameter.

Review Comment:
   If I hadn't changed `OmMetadataManagerImpl` constructor I might have taken this "lazy" approach. IMO that doesn't seem to be a memory-efficient way to pass along something that could be retrieved from the active OM. It is occupying 64 bits for each such request that are not flushed in the double buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240025228


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -246,7 +248,7 @@ public class OmMetadataManagerImpl implements OMMetadataManager,
 
   private DBStore store;
 
-  private final IOzoneManagerLock lock;
+  private IOzoneManagerLock lock;

Review Comment:
   
   
   
   > There are 40-ish places that calls the old constructor. I could fix that instead.
   
   Not necessary, I just wanted to understand.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1604754672

   Hey @smengcl It looks like this comment got marked as resolved just as I was responding to it: https://github.com/apache/ozone/pull/4567#discussion_r1240203895
   
   Please take another look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1244618933


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {
+      if (v == null) {
+        throw new IllegalArgumentException(
+            "Key '" + key + "' does not exist in cache");
       }
-    }
+
+      if (v.decrementRefCount() == 0L) {
+        synchronized (pendingEvictionList) {
+          // Eligible to be closed, add it to the list.
+          pendingEvictionList.add(v);
+          // The cache size might have already exceed the soft limit
+          // Thus triggering cleanup() to check and evict if applicable
+          cleanup();
+        }
+      }
+
+      return v;

Review Comment:
   nvm. turns out it is a bad idea to update other mappings inside a `map.compute()` anyways according to its javadoc. In this case it corrupts the map size count, and is caught by my UT:
   
   https://github.com/apache/ozone/actions/runs/5396424029/jobs/9800041580?pr=4567#step:6:2594
   
   Repro snippet:
   
   ```java
   import java.util.concurrent.ConcurrentHashMap;
   
   class Untitled {
     public static void main(String[] args) {
       ConcurrentHashMap map = new ConcurrentHashMap<String, String>();
       String k1 = "k1";
       map.put(k1, "v1");
       map.put("k2", "v2");
       System.out.println("- Map dump 1");
       map.forEach((k, v) -> {
         System.out.format("k=%s, v=%s\n", k, v, map.size());
       });
       System.out.println();
       
       System.out.println("- compute()");
       map.compute(k1, (k, v) -> {
         System.out.format("k=%s, v=%s, size=%d\n", k, v, map.size());
         map.remove(k);
         System.out.format("k=%s, v=%s, size=%d\n", k, v, map.size());
         return map.get(k);
       });
       System.out.println();
       // Map size becomes incorrect at this point
       System.out.format("k=%s, v=%s, size=%d\n", k1, map.get(k1), map.size());
       
       System.out.println("- Map dump 2");
       map.forEach((k, v) -> {
         System.out.format("k=%s, v=%s\n", k, v, map.size());
       });
     }
   }
   ```
   
   Output:
   ```
   - Map dump 1
   k=k1, v=v1
   k=k2, v=v2
   
   - compute()
   k=k1, v=v1, size=2
   k=k1, v=v1, size=1
   
   k=k1, v=null, size=0
   - Map dump 2
   k=k2, v=v2
   ```
   
   You can see map's `size=0` while it obviously still has `k2` entry in it.
   
   I have moved `cleanup()` out of `compute()` and added additional syncing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1220459396


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   Hi @GeorgeJahad , this method is not directly called from prod code anymore because we use `try-with-resources` now. It is only called internally. I think I can keep it here since `LoadingCache` has it. (Or mark it private)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1220785516


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   Maybe add a close() to SnapshotCache and make ReferenceCounted<> only take closeables?:
   ```
   public class ReferenceCounted<T implements Closeable> implements AutoCloseable {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1535481836

   > > @hemantk-12
   > > > I don't see cacheSize is used in eviction policy if cache is full. Yes, it doesn't proactively claims some entries but only if they are not used by anyone. If it need bounded, cache it doesn't solve that problem.
   > > 
   > > 
   > > It does seem to be implemented here, or am I misunderstanding your point?
   > > https://github.com/apache/ozone/blob/ddfc4b738d6d48ef37f3af41a7f743247933e30a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java#L167-L174
   > 
   > I had the same impression when I look at it first time but it is not bounded because `cleanup()` is called only by `release()`.
   
   Thanks for the reminder. It needs to call `cleanup()` by the end of `get()` as well. I will push the change later. (Doesn't change the fact that the cache is still unbounded.)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1535591440

   @GeorgeJahad In the latest commit posted 8b57f1f , I wrapped `OmSnapshot` inside `ReferenceCounted<>` so it becomes `ReferenceCounted<OmSnapshot>` in the cache `dbMap`.
   
   The plan is to replace every single instance of `OmSnapshot` with `ReferenceCounted<OmSnapshot>` in the rest of Ozone code so that they could be auto-closed with try-with-resources. This would involve quite a few changes:
   
   1. `IOmMetadataReader` would need to be wrapped as `ReferenceCounted<IOmMetadataReader>` as well. (10 usages in 6 files)
   2. Every single call to `OmSnapshotManager#checkForSnapshot` would now need to be wrapped inside try-with-resources. (10 usages in 6 files excluding tests)
   3. Need to wrap active DB's `OmMetadataReader` in `ReferenceCounted<>` as well due to its usage in `checkForSnapshot()`, although reference counting wouldn't be useful to active OM DB.
   4. `snapshotCache.get()` usages would need to be checked now that it returns `ReferenceCounted<OmSnapshot>`. (5 usages in 2 files)
   5. SnapDiff might not be able to use try-with-resources. Rather it would need to call `snapshotCache.release()` manually when a SnapDiff job is done on both DB instances (`fromDB` and `toDB`. CMIIW @hemantk-12
   
   I will postpone further work on this until #4642 is merged to avoid stepped on each others' toes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225170674


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   > Is it possible that release can be called twice on the same key?
   
   Only when `get()` is also called twice (or more times) already. Otherwise it is unintended undefined behavior.
   
   One `release()` (or `close()`) has to paired with one previous `get()`. Calling `release()` twice with only one previous `get()` could result in referenced count being decremented to an undefined `-1` (I think my code would catch this in `decrementRefCount()`).
   
   Just like one `unlock()` would pair to one previous `lock()` for `ReentrantLock`.
   
   Also, the entry is not evicted as soon as `release()` is called in this case. It could be marked as "eligible to be evicted" by adding it to the `pendingEvictionList`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240123740


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   With this PR, actual cache structure is `dbMap` in custom `SnapshotCache`. It is just utilizing the existing `CacheLoader` as the loader. Can either keep or remove it when we decide whether we want to make the cache impl pluggable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1236008468


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   Ok I'm going to add a TODO on `removalListener`. So that in the follow-up PR we can decide whether to keep it (make cache interface pluggable), or remove it (make custom SnapshotCache the only solution).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232889107


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   ok, but now I'm not quite sure how the underlying cache is stored.  Does it have it's own eviction policy.
   
   Also, I find it confusing to have the removalListener implemented but not having it be used anywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231749005


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);
+      cleanup();
+    }
+  }
+
+  /**
+   * Alternatively, can release with OmSnapshot instance directly.
+   * @param omSnapshot OmSnapshot
+   */
+  public void release(OmSnapshot omSnapshot) {
+    final String key = omSnapshot.getSnapshotTableKey();
+    release(key);
+  }
+
+  /**
+   * Callback method used to enqueue or dequeue ReferenceCounted from
+   * pendingEvictionList.
+   * @param referenceCounted ReferenceCounted object
+   */
+  public void callback(ReferenceCounted referenceCounted) {
+    if (referenceCounted.getTotalRefCount() == 0L) {
+      // Reference count reaches zero, add to pendingEvictionList
+      Preconditions.checkState(!pendingEvictionList.contains(referenceCounted),
+          "SnapshotCache is inconsistent. Entry should not be in the "
+              + "pendingEvictionList when ref count just reached zero.");
+      pendingEvictionList.add(referenceCounted);
+    } else if (referenceCounted.getTotalRefCount() == 1L) {
+      pendingEvictionList.remove(referenceCounted);
+    }
+  }
+
+  /**
+   * If cache size exceeds soft limit, attempt to clean up and close the
+   * instances that has zero reference count.
+   * TODO: [SNAPSHOT] Add new ozone debug CLI command to trigger this directly.
+   */
+  private void cleanup() {
+    long numEntriesToEvict = (long) dbMap.size() - cacheSizeLimit;
+    while (pendingEvictionList.size() > 0 && numEntriesToEvict > 0L) {
+      // Get the first instance in the clean up list
+      ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+          pendingEvictionList.iterator().next();
+      OmSnapshot omSnapshot = (OmSnapshot) rcOmSnapshot.get();
+      LOG.debug("Evicting OmSnapshot instance {} with table key {}",
+          rcOmSnapshot, omSnapshot.getSnapshotTableKey());
+      // Sanity check
+      Preconditions.checkState(rcOmSnapshot.getTotalRefCount() == 0L,
+          "Illegal state: OmSnapshot reference count non-zero ("
+              + rcOmSnapshot.getTotalRefCount() + ") but shows up in the "
+              + "clean up list");
+
+      final String key = omSnapshot.getSnapshotTableKey();
+      final ReferenceCounted<IOmMetadataReader> result = dbMap.remove(key);
+      // Sanity check
+      Preconditions.checkState(rcOmSnapshot == result,
+          "Cache map entry removal failure. The cache is in an inconsistent "
+              + "state. Expected OmSnapshot instance: " + rcOmSnapshot
+              + ", actual: " + result);
+
+      pendingEvictionList.remove(result);
+
+      // Close the instance, which also closes its DB handle.
+      try {
+        ((OmSnapshot) rcOmSnapshot.get()).close();
+      } catch (IOException ex) {
+        throw new IllegalStateException("Error while closing snapshot DB", ex);
+      }
+
+      --numEntriesToEvict;
+    }
+
+    // Print warning message if actual cache size is exceeding the soft limit
+    // even after the cleanup procedure above.
+    if ((long) dbMap.size() > cacheSizeLimit) {
+      LOG.warn("Current snapshot cache size ({}) is exceeding configured "
+          + "soft-limit ({}) after possible evictions.",
+          dbMap.size(), cacheSizeLimit);
+
+      Preconditions.checkState(pendingEvictionList.size() == 0);

Review Comment:
   `cleanup()` calls are now wrapped inside `synchronized (pendingEvictionList)` by callers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232879072


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   > Would you like to file a new jira for this? 
   Nope, just wanted to clarify.  (I didn't remember us saying we would replace the other approach.)
   
   But now I'm confused about why we are using com.google.common.cache.CacheLoader and  com.google.common.cache.RemovalListener.  Do we still need those?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239108575


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotMoveDeletedKeysResponse.java:
##########
@@ -78,42 +86,74 @@ public OMSnapshotMoveDeletedKeysResponse(@Nonnull OMResponse omResponse) {
   protected void addToDBBatch(OMMetadataManager omMetadataManager,
       BatchOperation batchOperation) throws IOException {
 
-    if (nextSnapshot != null) {
-      RDBStore nextSnapshotStore =
-          (RDBStore) nextSnapshot.getMetadataManager().getStore();
-      // Init Batch Operation for snapshot db.
-      try (BatchOperation writeBatch = nextSnapshotStore.initBatchOperation()) {
-        processKeys(writeBatch, nextSnapshot.getMetadataManager());
-        processDirs(writeBatch, nextSnapshot.getMetadataManager());
-        nextSnapshotStore.commitBatchOperation(writeBatch);
-        nextSnapshotStore.getDb().flushWal(true);
-        nextSnapshotStore.getDb().flush();
+    // Note: a trick to get
+    // To do this properly, refactor OzoneManagerDoubleBuffer#addToBatch and
+    // add OmSnapshotManager as a parameter.

Review Comment:
   If I hadn't changed `OmMetadataManagerImpl` constructor I might have taken this "lazy" approach. IMO that doesn't seem to be a memory-efficient way to pass along something that could be retrieved from the active OM. It is occupying 64 bits for each request that are not flushed in the double buffer.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239071162


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    synchronized (pendingEvictionList) {
+      // Remove instance from clean up list when it exists.
+      pendingEvictionList.remove(rcOmSnapshot);
+      // Check if any entries can be cleaned up.
+      // At this point, cache size might temporarily exceed cacheSizeLimit
+      // even if there are entries that can be evicted, which is fine since it
+      // is a soft limit.
+      cleanup();
+    }
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      synchronized (pendingEvictionList) {

Review Comment:
   Done in 5da12ac5c0593c2a12fce367163a972b5d3f6de0



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1540696308

   @smengcl I took a quick look at ReferenceCounted.java and SnapshotCache.java.  They look good to me.  (The cleanup() method probably requires some synchronization.)  I'll take a closer look when the PR comes out of draft state.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1509194545

   I also think some of the snapshot user code should be modified to support try-with-resources.  For example, this code:  https://github.com/apache/ozone/blob/ddfc4b738d6d48ef37f3af41a7f743247933e30a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java#L83-L91
   
   does the checkForSnapshot() call and then passes a reference to the omSnapshot into the double buffer thread.
   
   I think those sorts of calls should be modified just to pass the snapshot key into the double buffer thread and have it get the snapshot from the cache, (with try-with-resources.)
   
   The way it is implemented now, the reference is passed between two threads, and try-with-resources requires it all happen in the same thread.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1515176187

   > @hemantk-12
   > 
   > > I don't see cacheSize is used in eviction policy if cache is full. Yes, it doesn't proactively claims some entries but only if they are not used by anyone. If it need bounded, cache it doesn't solve that problem.
   > 
   > It does seem to be implemented here, or am I misunderstanding your point?
   > 
   > https://github.com/apache/ozone/blob/ddfc4b738d6d48ef37f3af41a7f743247933e30a/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java#L167-L174
   
   
   I had the same impression when I look at it first time but it is not bounded because `cleanup()` is called only by `release()`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1222358724


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   To me, the object passed into the `ReferenceCount<>` constructor should have a notion of how to clean up after itself after the refcount goes to 0.
   
   That seems like a fundamental part of a generic reference counting algorithm.  It shouldn't know about the object itself, just that the object has a particular interface.
   
   I guess I'm basically going back to my original thought.  One possiblity is to add a default no-op close method to `IOmMetadataReader` interface.  Then have the `SnapshotCache.get()` subclass that to return an `IOmMetadataReader` object which delegates most calls to the internal `IOmMetadataReader` but also has a close method that calls this.release().
   
   And then make `ReferenceCounted<>` only take closeables like I mentioned earlier:
   ```
   public class ReferenceCounted<T implements Closeable> implements AutoCloseable {
   ```
   
   and invoke obj.close() from within the `ReferenceCounted::decrementRefCount`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1222358724


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   To me, the object passed into the `ReferenceCount<>` constructor should have a notion of how to clean up after itself after the refcount goes to 0.
   
   That seems like a fundamental part of a generic reference counting algorithm.  It shouldn't know about the object itself, just that the object has a particular interface.
   
   I guess I'm basically going back to my original thought.  One possiblity is to add a default no-op close method to `IOmMetadataReader` interface.  Then have the `SnapshotCache.get()` subclass that to return an `IOmMetadataReader` object which delegates most calls to the internal `IOmMetadataReader` but also has a close method that calls this.release().
   
   And then make `ReferenceCounted<>` only take closeables like I mentioned earlier:
   ```
   public class ReferenceCounted<T implements Closeable> implements AutoCloseable {
   ```
   
   and invoke obj.close() from within the `ReferenceCounted::close`
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1222200884


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   You are right, I wasn't thinking about it correctly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1220617530


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   Thanks @smengcl   I see that try-with-resources is calling ReferenceCounted::close().  My question is how does that invoke SnapshotCache::release()?  
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232617700


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   I thought that the plan was to maintain the softValues() as a backup, i.e. do both "approach-1" and "approach-2".   Is that no longer the case?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225170674


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   > Is it possible that release can be called twice on the same key?
   
   Only when `get()` is also called twice (or more times). Otherwise it is unintended undefined behavior.
   
   One `release()` (or `close()`) has to paired with one previous `get()`. Calling `release()` twice with only one previous `get()` would result in referenced count being decremented to an undefined `-1`.
   
   Just like one `unlock()` would pair to one previous `lock()` for `ReentrantLock`.
   
   Also, the entry is not evicted as soon as `release()` is called in this case. It could be marked as "eligible to be evicted" by adding it to the `pendingEvictionList`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225165561


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestSnapshotDeletingService.java:
##########
@@ -471,4 +471,3 @@ private boolean assertTableRowCount(int expectedCount,
     return count == expectedCount;
   }
 }
-

Review Comment:
   [It does](https://github.dev/apache/ozone/blob/c12239078e9e8799159f5a01ac418bda07b0dced/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestSnapshotDeletingService.java):
   
   <img width="462" alt="eofnewline" src="https://github.com/apache/ozone/assets/50227127/14122c0d-856d-4eb6-9a57-6d8972f77497">
   
   Otherwise it would show a symbol like this at EOF: https://stackoverflow.com/q/65300309
   
   You can say that is a GH PR diff display issue.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231730355


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }
+
+      return newVal;
+    });
+
+    return refCount.get();
+  }
+
+  public long decrementRefCount() {
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    Preconditions.checkState(threadMap.containsKey(tid),
+        "Current thread have not holden reference before");
+
+    Preconditions.checkNotNull(threadMap.get(tid), "This thread " + tid +
+        " has not incremented the reference count before.");

Review Comment:
   Yeah the checks are there just to be paranoid.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231763014


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }

Review Comment:
   uh oh, findbugs is not happy with synchronizing on `AtomicLong`:
   
   ```
   M M JLM: Synchronization performed on java.util.concurrent.atomic.AtomicLong in org.apache.hadoop.ozone.om.snapshot.ReferenceCounted.decrementRefCount()  At ReferenceCounted.java:[line 118]
   M M JLM: Synchronization performed on java.util.concurrent.atomic.AtomicLong in org.apache.hadoop.ozone.om.snapshot.ReferenceCounted.incrementRefCount()  At ReferenceCounted.java:[line 83]
   ```
   
   https://github.com/apache/ozone/actions/runs/5286056671/jobs/9565123995



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231728514


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;

Review Comment:
   done: c2746dca3b4c21122e1cdb32b8004294ae729e6e
   
   The annoyance of `ReferenceCounted<IOmMetadataReader, SnapshotCache>` alone taking 50 chars with a 80-char line limit was one of the reasons I didn't do it before :)
   
   And `SnapshotCache` import is now everywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239072687


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -246,7 +248,7 @@ public class OmMetadataManagerImpl implements OMMetadataManager,
 
   private DBStore store;
 
-  private final IOzoneManagerLock lock;
+  private IOzoneManagerLock lock;

Review Comment:
   Because I added a new `OmMetadataManagerImpl` constructor and moved common logic to `init()` method where `lock` is then assigned.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240123740


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   With this PR, actual cache structure is `dbMap` in custom `SnapshotCache`. It is just utilizing the existing `CacheLoader` as the loader class. Can either keep or remove it when we decide whether we want to make the cache impl pluggable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1237616694


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -565,6 +564,8 @@ private synchronized SnapshotDiffResponse submitSnapDiffJob(
           volumeName, bucketName, fromSnapshotName, toSnapshotName,
           forceFullDiff));
       updateJobStatus(jobKey, QUEUED, IN_PROGRESS);
+      // DO NOT decrement snapshot instance ref count, executor is responsible

Review Comment:
   Ah. This is a left-over comment that belongs to the old approach before try-with-resources. Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1244630006


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {
+      if (v == null) {
+        throw new IllegalArgumentException(
+            "Key '" + key + "' does not exist in cache");
       }
-    }
+
+      if (v.decrementRefCount() == 0L) {
+        synchronized (pendingEvictionList) {
+          // Eligible to be closed, add it to the list.
+          pendingEvictionList.add(v);
+          // The cache size might have already exceed the soft limit
+          // Thus triggering cleanup() to check and evict if applicable
+          cleanup();
+        }
+      }
+
+      return v;

Review Comment:
   Since the ref counting is still done inside dbMap's monitor lock, the previously mentioned issue [here](https://github.com/apache/ozone/pull/4567#discussion_r1232881842) and [here](https://github.com/apache/ozone/pull/4567#discussion_r1232883122) should not revive.
   
   And `cleanup()` is now wrapped inside `synchronized`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234555540


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotDiffManager.java:
##########
@@ -565,6 +564,8 @@ private synchronized SnapshotDiffResponse submitSnapDiffJob(
           volumeName, bucketName, fromSnapshotName, toSnapshotName,
           forceFullDiff));
       updateJobStatus(jobKey, QUEUED, IN_PROGRESS);
+      // DO NOT decrement snapshot instance ref count, executor is responsible

Review Comment:
   This comment confuses me.  Which snapshot instance is being referred to here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231716860


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,

Review Comment:
   The reason I added this is the concern of atomic counting performance penalty.
   
   I think we could enable and disable counter in a perf test and see if / by how much will enabling this ref counting slow OM R/W down. ([currently](https://medium.com/engineering-cloudera/ozone-performance-improvements-c0bb8a76dc94#:~:text=above%20helped%20Ozone%E2%80%99s-,read%20key,-performance%20to%20go) OM key read >100k/s).
   
   Can remove this later if this is not the bottleneck



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225195161


##########
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestSnapshotDeletingService.java:
##########
@@ -471,4 +471,3 @@ private boolean assertTableRowCount(int expectedCount,
     return count == expectedCount;
   }
 }
-

Review Comment:
   Ah, I see, my bad. I think you removed an extra line and got confused. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239066138


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();

Review Comment:
   Good catch. I've changed `dbMap.computeIfAbsent()` into `dbMap.compute()` and increment ref count inside the atomic lambda block to patch this hole.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239073178


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmMetadataManagerImpl.java:
##########
@@ -246,7 +248,7 @@ public class OmMetadataManagerImpl implements OMMetadataManager,
 
   private DBStore store;
 
-  private final IOzoneManagerLock lock;
+  private IOzoneManagerLock lock;

Review Comment:
   There are 40-ish places that calls the old constructor. I could fix that instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240026165


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/snapshot/OMSnapshotMoveDeletedKeysResponse.java:
##########
@@ -78,42 +86,74 @@ public OMSnapshotMoveDeletedKeysResponse(@Nonnull OMResponse omResponse) {
   protected void addToDBBatch(OMMetadataManager omMetadataManager,
       BatchOperation batchOperation) throws IOException {
 
-    if (nextSnapshot != null) {
-      RDBStore nextSnapshotStore =
-          (RDBStore) nextSnapshot.getMetadataManager().getStore();
-      // Init Batch Operation for snapshot db.
-      try (BatchOperation writeBatch = nextSnapshotStore.initBatchOperation()) {
-        processKeys(writeBatch, nextSnapshot.getMetadataManager());
-        processDirs(writeBatch, nextSnapshot.getMetadataManager());
-        nextSnapshotStore.commitBatchOperation(writeBatch);
-        nextSnapshotStore.getDb().flushWal(true);
-        nextSnapshotStore.getDb().flush();
+    // Note: a trick to get
+    // To do this properly, refactor OzoneManagerDoubleBuffer#addToBatch and
+    // add OmSnapshotManager as a parameter.

Review Comment:
   ok



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225170674


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,

Review Comment:
   > Is it possible that release can be called twice on the same key?
   
   Only when `get()` is also called twice (or more times). Otherwise it is unintended undefined behavior.
   
   One `release()` (or `close()`) has to paired with one previous `get()`. Calling `release()` twice with only one previous `get()` could result in referenced count being decremented to an undefined `-1` (I think my code would catch this in `decrementRefCount()`).
   
   Just like one `unlock()` would pair to one previous `lock()` for `ReentrantLock`.
   
   Also, the entry is not evicted as soon as `release()` is called in this case. It could be marked as "eligible to be evicted" by adding it to the `pendingEvictionList`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232881842


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();

Review Comment:
   It seems like a problem that the cache load and the ref count inc are not synchronized.
   
   Suppose thread 1 has a ref to snap1 and thread 2 is running get() on snap1.
   
   Then, when thread 2 has passed over the computeIfAbsent() but before the count is incremented, thread 1 swaps in and releases snap1, lowers the count to 0 and cleansup the cache.
   
   Then thread 1 comes back and increments the count.
   
   Now the ref count is 1, but the snapshot isn't in memory at all.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232886229


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   > But now I'm confused about why we are using com.google.common.cache.CacheLoader and com.google.common.cache.RemovalListener. Do we still need those?
   
   We don't exactly need those anymore. I kept them from the beginning just because it would be less disruptive (less conflicts) if others are also fixing something in the loader at the same time. But this also leaves us the option to implement a pluggable interface if we need to.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231748728


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);
+
+    // Update the value and do some checks, atomically
+    threadMap.computeIfPresent(tid, (k, v) -> {
+      long newVal = v + 1;
+      Preconditions.checkState(newVal > 0L, "Thread reference count overflown");
+
+      long newValTotal = refCount.incrementAndGet();
+      Preconditions.checkState(newValTotal > 0L,
+          "Total reference count overflown");
+
+      if (refCount.get() == 1L) {
+        // ref count increased to one (from zero), remove from
+        // pendingEvictionList if added
+        parentSnapshotCache.callback(this);
+      }

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1242853326


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {
+      if (v == null) {
+        throw new IllegalArgumentException(
+            "Key '" + key + "' does not exist in cache");
       }
-    }
+
+      if (v.decrementRefCount() == 0L) {
+        synchronized (pendingEvictionList) {
+          // Eligible to be closed, add it to the list.
+          pendingEvictionList.add(v);
+          // The cache size might have already exceed the soft limit
+          // Thus triggering cleanup() to check and evict if applicable
+          cleanup();
+        }
+      }
+
+      return v;

Review Comment:
   Return v here will put it back in the dbMap, but if it gets removed in cleanup() above, I don' t think that is what we want.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1242850296


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {

Review Comment:
   the cleanup() call below calls dbMap.remove().   can that deadlock with this call to dbMap.compute()?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1244561291


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -224,29 +225,33 @@ public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
    * @param key snapshot table key
    */
   public void release(String key) {
-    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
-        dbMap.get(key);
-    Preconditions.checkNotNull(rcOmSnapshot,
-        "Key '" + key + "' does not exist in cache");
-
-    if (rcOmSnapshot.decrementRefCount() == 0L) {
-      synchronized (pendingEvictionList) {
-        // Eligible to be closed, add it to the list.
-        pendingEvictionList.add(rcOmSnapshot);
-        // The cache size might have already exceed the soft limit
-        // Thus triggering cleanup() to check and evict if applicable
-        cleanup();
+    dbMap.compute(key, (k, v) -> {

Review Comment:
   `synchronized` should be reentrant, meaning the same thread that already holds the object intrinsic lock can grab it again. So this should not be an issue in theory.
   
   https://docs.oracle.com/javase/tutorial/essential/concurrency/locksync.html
   
   https://stackoverflow.com/a/249907



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240392668


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    synchronized (pendingEvictionList) {
+      // Remove instance from clean up list when it exists.
+      pendingEvictionList.remove(rcOmSnapshot);
+      // Check if any entries can be cleaned up.
+      // At this point, cache size might temporarily exceed cacheSizeLimit
+      // even if there are entries that can be evicted, which is fine since it
+      // is a soft limit.
+      cleanup();
+    }
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      synchronized (pendingEvictionList) {

Review Comment:
   I see. That could indeed be an issue.
   
   > And moving the "if 0" test into the synchronized block won't help, because increment() isn't protected by that lock. So the increment() could happen anytime during the synchronized block.
   
   Putting `decrementRefCount()` inside a `synchronized (dbMap)` should work now that 5da12ac5c0593c2a12fce367163a972b5d3f6de0 is in. Because:
   
   1. `incrementRefCount()` is now only done in the `compute()` lambda.
   2. `dbMap.compute()` effectively runs the lambda inside a `synchronized (dbMap)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1563360840

   This PR is ready for review at this point.
   
   The overall structure would not receive any big changes from me without further comments.
   
   I will mark the PR ready for review once I resolve the test timeout (caused by strict thread ID checking in `ReferenceCounted`) in a bit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1207398767


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   I only see this method being called in the tests.  Doesn't it need to be called from the real code somewhere?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] hemantk-12 commented on pull request #4567: HDDS-7935. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["hemantk-12 (via GitHub)" <gi...@apache.org>]

hemantk-12 commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1509349573

   Only for the snapshot use case, I am inclined towards approach #2.
   
   Reason: 
   1. The way [SnapshotCache](https://github.com/apache/ozone/pull/4567/files#diff-d5b0b24274c5d19e876dc2ffeb744d4965fa1a23ce727ac09b965f665573c18aR38) is implemented is not bounded cache. I don't see [cacheSize](https://github.com/apache/ozone/pull/4567/files#diff-d5b0b24274c5d19e876dc2ffeb744d4965fa1a23ce727ac09b965f665573c18aR55) is used in eviction policy if cache is full. Yes, it doesn't proactively claims some entries but only if they are not used by anyone. If it was bounded cache I would have gone the first approach.
   2. If I understood it properly, Cache is used in snapshot to reuse the RocksDB instance instead of create a new one every time. In that case, I think it is OK to rely on GC for that.
   
   I was thinking to use [WeakReference](https://docs.oracle.com/javase/8/docs/api/java/lang/ref/WeakReference.html) with [WeakHashMap](https://docs.oracle.com/javase/7/docs/api/java/util/WeakHashMap.html) which I guess would be similar to `CacheBuilder#weakValues()` and `CacheBuilder#softValues()`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1239092311


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMDirectoriesPurgeRequestWithFSO.java:
##########
@@ -148,7 +151,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
         getOmRequest());
     OMClientResponse omClientResponse = new OMDirectoriesPurgeResponseWithFSO(
         omResponse.build(), purgeRequests, ozoneManager.isRatisEnabled(),
-            getBucketLayout(), volBucketInfoMap, omFromSnapshot);
+            getBucketLayout(), volBucketInfoMap, rcOmFromSnapshot);

Review Comment:
   Yep you are right I will need to do that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1234543967


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMDirectoriesPurgeRequestWithFSO.java:
##########
@@ -148,7 +151,7 @@ public OMClientResponse validateAndUpdateCache(OzoneManager ozoneManager,
         getOmRequest());
     OMClientResponse omClientResponse = new OMDirectoriesPurgeResponseWithFSO(
         omResponse.build(), purgeRequests, ozoneManager.isRatisEnabled(),
-            getBucketLayout(), volBucketInfoMap, omFromSnapshot);
+            getBucketLayout(), volBucketInfoMap, rcOmFromSnapshot);

Review Comment:
   In OMKeyPurgeRequest.java, you changed the purgeResponse parameter from snapshot to snapshotInfo.  Do we want to do the same thing here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1232883122


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    synchronized (pendingEvictionList) {
+      // Remove instance from clean up list when it exists.
+      pendingEvictionList.remove(rcOmSnapshot);
+      // Check if any entries can be cleaned up.
+      // At this point, cache size might temporarily exceed cacheSizeLimit
+      // even if there are entries that can be evicted, which is fine since it
+      // is a soft limit.
+      cleanup();
+    }
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      synchronized (pendingEvictionList) {

Review Comment:
   If another thread does a get() between lines 228 and 229, it will increment the count and we will still remove the snapshot below.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240123740


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OmSnapshotManager.java:
##########
@@ -246,15 +246,8 @@ public OmSnapshotManager(OzoneManager ozoneManager) {
       }
     };
 
-    // init LRU cache
-    this.snapshotCache = CacheBuilder.newBuilder()
-        // Indicating OmSnapshot instances are softly referenced from the cache.
-        // If no thread is holding a strong reference to an OmSnapshot instance
-        // (e.g. SnapDiff), the instance could be garbage collected by JVM at
-        // its discretion.
-        .softValues()

Review Comment:
   With this PR, actual cache structure is `dbMap` in custom `SnapshotCache`. It is just utilizing the existing `CacheLoader` as the loader class. Can either keep or remove it when depending on whether we want to make the cache impl pluggable in a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1240203895


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,375 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache implements ReferenceCountedCallback {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: [SNAPSHOT] Consider wrapping SoftReference<> around IOmMetadataReader
+  private final ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  private final Set<
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList =
+        Collections.synchronizedSet(new LinkedHashSet<>());
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String,
+      ReferenceCounted<IOmMetadataReader, SnapshotCache>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  Set<ReferenceCounted<
+      IOmMetadataReader, SnapshotCache>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<
+        Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader, SnapshotCache>>
+          entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader, SnapshotCache> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active anymore.
+    if (!skipActiveCheck &&
+        !omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE)) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    synchronized (pendingEvictionList) {
+      // Remove instance from clean up list when it exists.
+      pendingEvictionList.remove(rcOmSnapshot);
+      // Check if any entries can be cleaned up.
+      // At this point, cache size might temporarily exceed cacheSizeLimit
+      // even if there are entries that can be evicted, which is fine since it
+      // is a soft limit.
+      cleanup();
+    }
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader, SnapshotCache> rcOmSnapshot =
+        dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      synchronized (pendingEvictionList) {

Review Comment:
   
   > I believe you mean "it will decrement the count"?
   
   No I meant the other thread doing the get() will increment the count.
   
   Let me try to clarify.  Suppose thread 1 is doing a release(). It runs the "if 0" test above, finds the count is 0, and starts to enter the sychronized block.  But the lock is taken so it waits.  Meanwhile thread 2 takes control and does a get().
   
   Now the reference count is 1, and control returns to thread 1.  That thread now enters the synchronized block, adds the snapshot to the eviction list, and does a cleanup() which could evict the snapshot from the cache.  But the refcount is still 1, and thread 2 is still going to try to use the snapshot without reloading it into the cache.
   
   Isn't that possible?
   
   And moving the "if 0" test into the synchronized block won't help, because increment() isn't protected by that lock.  So the increment() could happen anytime during the synchronized block.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on PR #4567:
URL: https://github.com/apache/ozone/pull/4567#issuecomment-1612155385

   Thanks @GeorgeJahad and @hemantk-12 for reviewing this! I will merge this shortly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1220815210


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   ah close() doesn't take a parmeter, so it would need to be a different interface, but same principle.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1223096661


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+  /**
+   * Get or load OmSnapshot. Must be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k));
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !SnapshotUtils.isCalledFromSnapshotDeletingService()) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {

Review Comment:
   Actually, the `OzoneManager`  isalso an `IOmMetadataReader` that implements close(), which we don't want to accidentally invoke here.  So instead of using the `Closable` interface here, I would probably create a new different interface, "Releaseable" and have `ReferenceCounted` take that:
   ```
   public class ReferenceCounted<T implements Releaseable> implements AutoCloseable {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1225174130


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically

Review Comment:
   The emphasis of this comment is on this operation intending to be _atomic_, and to prompt reviewers to check if this is indeed _atomic_.
   
   It has achieved its purpose. I can remove it later.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231714733


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;
+
+  public ReferenceCounted(T obj, boolean disableCounter,
+      SnapshotCache parentSnapshotCache) {
+    // A param to allow disabling ref counting to reduce active DB
+    //  access penalties due to AtomicLong operations.
+    this.obj = obj;
+    if (disableCounter) {
+      this.threadMap = null;
+      this.refCount = null;
+    } else {
+      this.threadMap = new ConcurrentHashMap<>();
+      this.refCount = new AtomicLong(0L);
+    }
+    this.parentSnapshotCache = parentSnapshotCache;
+  }
+
+  /**
+   * @return Object being referenced counted.
+   */
+  public T get() {
+    return obj;
+  }
+
+  public long incrementRefCount() { // TODO: [SNAPSHOT] Rename to increment()
+    if (refCount == null) {
+      return -1L;
+    }
+
+    long tid = Thread.currentThread().getId();
+
+    // Put the new mapping if absent, atomically
+    threadMap.putIfAbsent(tid, 0L);

Review Comment:
   if `threadMap` is null, `refCount` must have been null as well, which falls into the existing check.
   
   adding the extra check to make intellij happy.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231728514


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;
+
+  /**
+   * Sum of reference counts from all threads.
+   */
+  private final AtomicLong refCount;
+
+  /**
+   * Parent SnapshotCache instance whose callback will be triggered upon this RC
+   * closure.
+   */
+  private final SnapshotCache parentSnapshotCache;

Review Comment:
   done: c2746dca3b4c21122e1cdb32b8004294ae729e6e
   
   The annoyance of `ReferenceCounted<IOmMetadataReader, SnapshotCache>` alone taking 50 chars with a 80-char line limit was one of the reasons I didn't do it before :)
   
   And `SnapshotCache` imports are now everywhere.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] smengcl commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["smengcl (via GitHub)" <gi...@apache.org>]

smengcl commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1231734805


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/ReferenceCounted.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.base.Preconditions;
+
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.atomic.AtomicLong;
+
+/**
+ * Add reference counter to an object instance.
+ */
+public class ReferenceCounted<T> implements AutoCloseable {
+
+  /**
+   * Object that is being reference counted. e.g. OmSnapshot
+   */
+  private final T obj;
+
+  /**
+   * A map of thread IDs holding the reference of the object and its count.
+   */
+  private final ConcurrentHashMap<Long, Long> threadMap;

Review Comment:
   Initially I was also thinking of using [the new enum](https://github.com/apache/ozone/blob/c2746dca3b4c21122e1cdb32b8004294ae729e6e/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java#L136-L144) to describe what each thread is grabbing the cache instance for. So that when there are leaks I can trace back to which thread and which usage.
   
   I'd have to further change the value of `threadMap` in order to achieve that. Or it would be much easier to just print debug logs.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org

[GitHub] [ozone] GeorgeJahad commented on a diff in pull request #4567: HDDS-8528. [Snapshot] Custom SnapshotCache implementation to replace LoadingCache

["GeorgeJahad (via GitHub)" <gi...@apache.org>]

GeorgeJahad commented on code in PR #4567:
URL: https://github.com/apache/ozone/pull/4567#discussion_r1230267684


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/snapshot/SnapshotCache.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ *  with the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+package org.apache.hadoop.ozone.om.snapshot;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.cache.CacheLoader;
+import org.apache.hadoop.ozone.om.IOmMetadataReader;
+import org.apache.hadoop.ozone.om.OmSnapshot;
+import org.apache.hadoop.ozone.om.OmSnapshotManager;
+import org.apache.hadoop.ozone.om.exceptions.OMException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import static org.apache.hadoop.ozone.om.exceptions.OMException.ResultCodes.FILE_NOT_FOUND;
+import static org.apache.hadoop.ozone.om.helpers.SnapshotInfo.SnapshotStatus.SNAPSHOT_ACTIVE;
+
+/**
+ * Thread-safe custom unbounded LRU cache to manage open snapshot DB instances.
+ */
+public class SnapshotCache {
+
+  static final Logger LOG = LoggerFactory.getLogger(SnapshotCache.class);
+
+  // Snapshot cache internal hash map.
+  // Key:   DB snapshot table key
+  // Value: OmSnapshot instance, each holds a DB instance handle inside
+  // TODO: Also wrap SoftReference<> around the value?
+  private final ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>>
+      dbMap;
+
+  // Linked hash set that holds OmSnapshot instances whose reference count
+  // has reached zero. Those entries are eligible to be evicted and closed.
+  // Sorted in last used order.
+  // Least-recently-used entry located at the beginning.
+  // TODO: Check thread safety. Try ConcurrentHashMultiset ?
+  private final LinkedHashSet<ReferenceCounted<IOmMetadataReader>>
+      pendingEvictionList;
+  private final OmSnapshotManager omSnapshotManager;
+  private final CacheLoader<String, OmSnapshot> cacheLoader;
+  // Soft-limit of the total number of snapshot DB instances allowed to be
+  // opened on the OM.
+  private final int cacheSizeLimit;
+
+  public SnapshotCache(
+      OmSnapshotManager omSnapshotManager,
+      CacheLoader<String, OmSnapshot> cacheLoader,
+      int cacheSizeLimit) {
+    this.dbMap = new ConcurrentHashMap<>();
+    this.pendingEvictionList = new LinkedHashSet<>();
+    this.omSnapshotManager = omSnapshotManager;
+    this.cacheLoader = cacheLoader;
+    this.cacheSizeLimit = cacheSizeLimit;
+  }
+
+  @VisibleForTesting
+  ConcurrentHashMap<String, ReferenceCounted<IOmMetadataReader>> getDbMap() {
+    return dbMap;
+  }
+
+  @VisibleForTesting
+  LinkedHashSet<ReferenceCounted<IOmMetadataReader>> getPendingEvictionList() {
+    return pendingEvictionList;
+  }
+
+  /**
+   * @return number of DB instances currently held in cache.
+   */
+  public int size() {
+    return dbMap.size();
+  }
+
+  /**
+   * Immediately invalidate an entry.
+   * @param key DB snapshot table key
+   */
+  public void invalidate(String key) throws IOException {
+    dbMap.computeIfPresent(key, (k, v) -> {
+      pendingEvictionList.remove(v);
+      try {
+        ((OmSnapshot) v.get()).close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot: " + key, e);
+      }
+      // Remove the entry from map by returning null
+      return null;
+    });
+  }
+
+  /**
+   * Immediately invalidate all entries and close their DB instances in cache.
+   */
+  public void invalidateAll() {
+    Iterator<Map.Entry<String, ReferenceCounted<IOmMetadataReader>>>
+        it = dbMap.entrySet().iterator();
+
+    while (it.hasNext()) {
+      Map.Entry<String, ReferenceCounted<IOmMetadataReader>> entry = it.next();
+      pendingEvictionList.remove(entry.getValue());
+      OmSnapshot omSnapshot = (OmSnapshot) entry.getValue().get();
+      try {
+        // TODO: If wrapped with SoftReference<>, omSnapshot could be null?
+        omSnapshot.close();
+      } catch (IOException e) {
+        throw new IllegalStateException("Failed to close snapshot", e);
+      }
+      it.remove();
+    }
+  }
+
+  /**
+   * State the reason the current thread is getting the OmSnapshot instance.
+   */
+  public enum Reason {
+    FS_API_READ,
+    SNAPDIFF_READ,
+    DEEP_CLEAN_WRITE,
+    GARBAGE_COLLECTION_WRITE
+  }
+
+
+  public ReferenceCounted<IOmMetadataReader> get(String key)
+      throws IOException {
+    return get(key, false);
+  }
+
+  /**
+   * Get or load OmSnapshot. Shall be close()d after use.
+   * TODO: [SNAPSHOT] Can add reason enum to param list later.
+   * @param key snapshot table key
+   * @return an OmSnapshot instance, or null on error
+   */
+  public ReferenceCounted<IOmMetadataReader> get(String key,
+      boolean skipActiveCheck) throws IOException {
+    // Atomic operation to initialize the OmSnapshot instance (once) if the key
+    // does not exist.
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot =
+        dbMap.computeIfAbsent(key, k -> {
+          LOG.info("Loading snapshot. Table key: {}", k);
+          try {
+            return new ReferenceCounted<>(cacheLoader.load(k), false, this);
+          } catch (OMException omEx) {
+            // Return null if the snapshot is no longer active
+            if (!omEx.getResult().equals(FILE_NOT_FOUND)) {
+              throw new IllegalStateException(omEx);
+            }
+          } catch (IOException ioEx) {
+            // Failed to load snapshot DB
+            throw new IllegalStateException(ioEx);
+          } catch (Exception ex) {
+            // Unexpected and unknown exception thrown from CacheLoader#load
+            throw new IllegalStateException(ex);
+          }
+          // Do not put the value in the map on exception
+          return null;
+        });
+
+    if (rcOmSnapshot == null) {
+      // The only exception that would fall through the loader logic above
+      // is OMException with FILE_NOT_FOUND.
+      throw new OMException("Snapshot table key '" + key + "' not found, "
+          + "or the snapshot is no longer active",
+          OMException.ResultCodes.FILE_NOT_FOUND);
+    }
+
+    // If the snapshot is already loaded in cache, the check inside the loader
+    // above is ignored. But we would still want to reject all get()s except
+    // when called from SDT (and some) if the snapshot is not active any more.
+    if (!omSnapshotManager.isSnapshotStatus(key, SNAPSHOT_ACTIVE) &&
+        !skipActiveCheck) {
+      throw new OMException("Unable to load snapshot. " +
+          "Snapshot with table key '" + key + "' is no longer active",
+          FILE_NOT_FOUND);
+    }
+
+    // Increment the reference count on the instance.
+    rcOmSnapshot.incrementRefCount();
+
+    // Remove instance from clean up list when it exists.
+    // TODO: [SNAPSHOT] Check thread safety with release()
+    pendingEvictionList.remove(rcOmSnapshot);
+
+    // Check if any entries can be cleaned up.
+    // At this point, cache size might temporarily exceed cacheSizeLimit
+    // even if there are entries that can be evicted, which is fine since it
+    // is a soft limit.
+    cleanup();
+
+    return rcOmSnapshot;
+  }
+
+  /**
+   * Release the reference count on the OmSnapshot instance.
+   * @param key snapshot table key
+   */
+  public void release(String key) {
+    ReferenceCounted<IOmMetadataReader> rcOmSnapshot = dbMap.get(key);
+    Preconditions.checkNotNull(rcOmSnapshot,
+        "Key '" + key + "' does not exist in cache");
+
+    if (rcOmSnapshot.decrementRefCount() == 0L) {
+      // Eligible to be closed, add it to the list.
+      pendingEvictionList.add(rcOmSnapshot);

Review Comment:
   I think the refcount value check has to synchronized with the evictionList mods; otherwise some other thread can modify the refcount in between the two.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org