[GitHub] [pulsar] eolivelli commented on a change in pull request #14930: Offloader: add API to scan objects on Tiered Storage

[GitBox <gi...@apache.org>]

eolivelli commented on a change in pull request #14930:
URL: https://github.com/apache/pulsar/pull/14930#discussion_r839369130



##########
File path: tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreManagedLedgerOffloader.java
##########
@@ -621,4 +634,75 @@ public void close() {
             }
         }
     }
+
+    @Override
+    public void scanLedgers(OffloadedLedgerMetadataConsumer consumer, Map<String, String> offloadDriverMetadata) throws ManagedLedgerException {
+        BlobStoreLocation bsKey = getBlobStoreLocation(offloadDriverMetadata);
+        String endpoint = bsKey.getEndpoint();
+        String readBucket = bsKey.getBucket();
+        log.info("Scanning bucket {}, bsKey {}, location {} endpoint{} ", readBucket, bsKey,
+                config.getBlobStoreLocation(), endpoint);
+        BlobStore readBlobstore = blobStores.get(config.getBlobStoreLocation());
+        int batchSize = 100;
+        String bucketName = config.getBucket();
+        String marker = null;
+        do {
+            marker = scanContainer(consumer, readBlobstore, bucketName, marker, batchSize);
+        } while (marker != null);
+
+    }
+
+    private String scanContainer(OffloadedLedgerMetadataConsumer consumer, BlobStore readBlobstore,
+                                 String bucketName,
+                                 String lastMarker,
+                                 int batchSize) throws ManagedLedgerException {
+        ListContainerOptions options = new ListContainerOptions()
+                .maxResults(batchSize)
+                .withDetails();
+        if (lastMarker != null) {
+            options.afterMarker(lastMarker);
+        }
+        PageSet<? extends StorageMetadata> pages = readBlobstore.list(bucketName, options);
+        for (StorageMetadata md : pages) {
+            log.info("Found {} ",md);
+            String name = md.getName();
+            Long size = md.getSize();
+            Date lastModified = md.getLastModified();
+            StorageType type = md.getType();
+            if (type != StorageType.BLOB) {
+                continue;
+            }
+            URI uri = md.getUri();
+            Map<String, String> userMetadata = md.getUserMetadata();

Review comment:
       in this patch I added that information
   https://github.com/apache/pulsar/pull/14907
   see here
   https://github.com/apache/pulsar/pull/14907/files#diff-10c82eb07e9dae9051b656297cf7dc38d051aad4b61b8555d076d56b4030b5bdR188




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org