[GitHub] [pulsar] dave2wave commented on a change in pull request #14694: [Issue 14693][Offloader] JCloud Offloader Now Removes S3 Objects

[GitBox <gi...@apache.org>]

dave2wave commented on a change in pull request #14694:
URL: https://github.com/apache/pulsar/pull/14694#discussion_r828115697



##########
File path: tiered-storage/jcloud/src/main/java/org/apache/bookkeeper/mledger/offload/jcloud/impl/BlobStoreManagedLedgerOffloader.java
##########
@@ -555,13 +555,18 @@ private BlobStoreLocation getBlobStoreLocation(Map<String, String> offloadDriver
         BlobStoreLocation bsKey = getBlobStoreLocation(offloadDriverMetadata);
         String readBucket = bsKey.getBucket(offloadDriverMetadata);
         BlobStore readBlobstore = blobStores.get(config.getBlobStoreLocation());
+        String dataContent = DataBlockUtils.dataBlockOffloadKey(ledgerId, uid);
+        String indexContent = DataBlockUtils.indexBlockOffloadKey(ledgerId, uid);
 
         CompletableFuture<Void> promise = new CompletableFuture<>();
         scheduler.chooseThread(ledgerId).submit(() -> {
             try {
-                readBlobstore.removeBlobs(readBucket,
-                    ImmutableList.of(DataBlockUtils.dataBlockOffloadKey(ledgerId, uid),
-                                     DataBlockUtils.indexBlockOffloadKey(ledgerId, uid)));

Review comment:
       Exactly, DeleteObjects fails without throwing an exception. Rather than speculate why I chose to try to see if two consecutive DeleteObject calls would work and they did.
   
   If I am speculating then I did have to explicitly set the S3 URL because I am in the `us-west-2` region instead of the default for S3 region of `us-east-1`. I wonder if the POST method has a bad header.
   
   I gather that this once worked and I looked at the JCloud code history.... but don't see anything of concern.
   
   While this fix may not be the "best" I think it is important to share since leaked S3 objects cost money and are hard to clean up.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org