You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sadanand48 (via GitHub)" <gi...@apache.org> on 2026/09/08 08:42:37 UTC

[PR] HDDS-16389. Add command to display snapshot counts per bucket. [ozone]

sadanand48 opened a new pull request, #11217:
URL: https://github.com/apache/ozone/pull/11217

   ## What changes were proposed in this pull request?
   This PR is to add a command to understand snapshot counts per bucket as currently there is no direct way to fetch this info
   
   ## What is the link to the Apache JIRA
   https://issues.apache.org/jira/browse/HDDS-16389
   
   ## How was this patch tested?
   Unit tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16389. Add command to display snapshot counts per bucket. [ozone]

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 merged PR #11217:
URL: https://github.com/apache/ozone/pull/11217


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16389. Add command to display snapshot counts per bucket. [ozone]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #11217:
URL: https://github.com/apache/ozone/pull/11217#discussion_r4006746338


##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -3268,6 +3270,144 @@ public ListSnapshotResponse listSnapshot(
     }
   }
 
+  @Override
+  public SnapshotCountResponse snapshotCount(String bucketFilter)
+      throws IOException {
+    Map<String, String> auditMap = new LinkedHashMap<>();
+    auditMap.put("bucketFilter", StringUtils.defaultString(bucketFilter));
+
+    try {
+      BucketFilter parsedFilter = BucketFilter.parse(bucketFilter);
+      Map<String, BucketSnapshotCount> bucketCounts = new LinkedHashMap<>();
+      Map<String, Boolean> bucketAclCache = new LinkedHashMap<>();
+
+      try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
+               metadataManager.getSnapshotInfoTable().iterator()) {
+        while (keyIter.hasNext()) {
+          SnapshotInfo snapshotInfo = keyIter.next().getValue();
+          String volumeName = snapshotInfo.getVolumeName();
+          String bucketName = snapshotInfo.getBucketName();
+          if (!parsedFilter.matches(volumeName, bucketName)) {
+            continue;
+          }
+
+          if (getAclsEnabled() && !hasListAccess(volumeName, bucketName, bucketAclCache)) {
+            continue;
+          }
+
+          String bucketKey = volumeName + "/" + bucketName;
+          BucketSnapshotCount bucketCount =
+              bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName));
+          bucketCount.increment(snapshotInfo.getSnapshotStatus());

Review Comment:
   The added handler test fabricates a zero-count bucket, but this implementation only creates a `BucketSnapshotCount` while iterating a `SnapshotInfo` row. An existing bucket with no snapshots has no such row, so `--bucket vol1/bucket2` returns an empty `buckets` object instead of the zero entry asserted by `testCountBucketFilterIncludesMatchingEmptyBucket`. Either seed the filtered existing bucket in the server response or align the test and documented contract.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -3268,6 +3270,144 @@ public ListSnapshotResponse listSnapshot(
     }
   }
 
+  @Override
+  public SnapshotCountResponse snapshotCount(String bucketFilter)
+      throws IOException {
+    Map<String, String> auditMap = new LinkedHashMap<>();
+    auditMap.put("bucketFilter", StringUtils.defaultString(bucketFilter));
+
+    try {
+      BucketFilter parsedFilter = BucketFilter.parse(bucketFilter);
+      Map<String, BucketSnapshotCount> bucketCounts = new LinkedHashMap<>();
+      Map<String, Boolean> bucketAclCache = new LinkedHashMap<>();
+
+      try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
+               metadataManager.getSnapshotInfoTable().iterator()) {
+        while (keyIter.hasNext()) {
+          SnapshotInfo snapshotInfo = keyIter.next().getValue();

Review Comment:
   This raw table iterator reads only persisted RocksDB rows and does not merge the table cache. Snapshot create/delete requests place the new or status-updated `SnapshotInfo` in that cache before the double buffer flushes it, while `OmMetadataManagerImpl.listSnapshot` uses the cache-aware `ListIterator.MinHeapIterator`; therefore this command can undercount new snapshots and report deleted snapshots as active until a flush. Use the cache-aware merge path (or explicitly merge/filter cache entries) here.



##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java:
##########
@@ -3268,6 +3270,144 @@ public ListSnapshotResponse listSnapshot(
     }
   }
 
+  @Override
+  public SnapshotCountResponse snapshotCount(String bucketFilter)
+      throws IOException {
+    Map<String, String> auditMap = new LinkedHashMap<>();
+    auditMap.put("bucketFilter", StringUtils.defaultString(bucketFilter));
+
+    try {
+      BucketFilter parsedFilter = BucketFilter.parse(bucketFilter);
+      Map<String, BucketSnapshotCount> bucketCounts = new LinkedHashMap<>();
+      Map<String, Boolean> bucketAclCache = new LinkedHashMap<>();
+
+      try (TableIterator<String, ? extends Table.KeyValue<String, SnapshotInfo>> keyIter =
+               metadataManager.getSnapshotInfoTable().iterator()) {
+        while (keyIter.hasNext()) {
+          SnapshotInfo snapshotInfo = keyIter.next().getValue();
+          String volumeName = snapshotInfo.getVolumeName();
+          String bucketName = snapshotInfo.getBucketName();
+          if (!parsedFilter.matches(volumeName, bucketName)) {
+            continue;
+          }
+
+          if (getAclsEnabled() && !hasListAccess(volumeName, bucketName, bucketAclCache)) {
+            continue;
+          }
+
+          String bucketKey = volumeName + "/" + bucketName;
+          BucketSnapshotCount bucketCount =
+              bucketCounts.computeIfAbsent(bucketKey, unused -> new BucketSnapshotCount(volumeName, bucketName));
+          bucketCount.increment(snapshotInfo.getSnapshotStatus());
+        }
+      }
+
+      long totalActive = 0;
+      long totalDeleted = 0;
+      long total = 0;
+      List<SnapshotBucketCount> bucketCountList = new ArrayList<>(bucketCounts.size());
+      for (BucketSnapshotCount bucketCount : bucketCounts.values()) {
+        bucketCountList.add(bucketCount.toResponse());
+        totalActive += bucketCount.active;
+        totalDeleted += bucketCount.deleted;
+        total += bucketCount.total;
+      }
+
+      AUDIT.logReadSuccess(buildAuditMessageForSuccess(OMAction.LIST_SNAPSHOT, auditMap));
+      return new SnapshotCountResponse(totalActive, totalDeleted, total, bucketCountList);
+    } catch (Exception ex) {
+      AUDIT.logReadFailure(buildAuditMessageForFailure(OMAction.LIST_SNAPSHOT, auditMap, ex));
+      throw ex;
+    }
+  }
+
+  private boolean hasListAccess(String volumeName, String bucketName, Map<String, Boolean> bucketAclCache)
+      throws IOException {
+    String bucketKey = volumeName + "/" + bucketName;
+    Boolean hasAccess = bucketAclCache.get(bucketKey);
+    if (hasAccess != null) {
+      return hasAccess;
+    }
+
+    try {
+      omMetadataReader.checkAcls(ResourceType.BUCKET, StoreType.OZONE, ACLType.LIST, volumeName, bucketName, null);
+      bucketAclCache.put(bucketKey, true);
+      return true;
+    } catch (OMException ex) {
+      if (ex.getResult() == OMException.ResultCodes.ACCESS_DENIED) {

Review Comment:
   `OmMetadataReader.checkAcls` throws `OMException.ResultCodes.PERMISSION_DENIED` when access is denied, not `ACCESS_DENIED` (see its implementation). With ACLs enabled, encountering the first bucket the caller cannot list therefore aborts the global count request instead of skipping that bucket. Catch the result code emitted by `checkAcls` here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org


Re: [PR] HDDS-16389. Add command to display snapshot counts per bucket. [ozone]

Posted by "sadanand48 (via GitHub)" <gi...@apache.org>.
sadanand48 commented on PR #11217:
URL: https://github.com/apache/ozone/pull/11217#issuecomment-5727395747

   Thanks @aryangupta1998 for the review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org