BlobStore list() and PageSet

[GARDAIS Ionel <io...@tech-advantage.com>]

Hi, 

I have a question regarding BlobStore listing. 
We are currently using the FileSystem storage implementation 

private long containerSize(String containerName) { 
long containerSize = 0; 
String marker = null; 
PageSet<? extends StorageMetadata> containerSML = null; 
ListContainerOptions lcoRecursive = new ListContainerOptions().recursive(); 

(1) containerSML = blobStore.list(containerName, lcoRecursive); 
containerSize += pageSize(containerSML); 
marker = containerSML.getNextMarker(); 

while (marker != null) { 
lcoRecursive.afterMarker(marker); 
(2) containerSML = blobStore.list(containerName, lcoRecursive); 
containerSize += pageSize(containerSML); 
marker = containerSML.getNextMarker(); 
} 

log.debug("container:{} size:{}", containerName, containerSize); 
return containerSize; 
} 
private long pageSize(PageSet<? extends StorageMetadata> containerStorageMetadataList) { 
long size = 0; 
for (StorageMetadata containerStorageMetadata : containerStorageMetadataList) { 
if (containerStorageMetadata.getType() == StorageType.BLOB) { 
log.debug("name:{} size:{}", containerStorageMetadata.getName(), containerStorageMetadata.getSize()); 
size += containerStorageMetadata.getSize(); 
} else { 
log.debug("other in container: {}", containerStorageMetadata.getType()); 
} 
} 
return size; 
} 


By calling list() with ListContainerOptions().recursive() on both (1) and (2), all container’s blobs are opened every PageSet. 
I tried to call list() with recursive() on (1) only then iterating over PageSets without the recursive flag set but I don’t get all Blobs. 

Is there a way to iterate over all blobs in a container in a more effective way ? 

Thanks, 
Ionel 

--

232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON

Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301

Re: BlobStore list() and PageSet

[Andrew Gaul <ga...@apache.org>]

Unfortunately both the transient and filesystem blobstores have
inefficient implementations which enumerate the entire blobstore even
when returning only a subset of keys.  LocalBlobStore.list calls
getBlobKeysInsideContainer and filters on the output.  Instead it should
call into the storage strategy with the marker, limit, prefix, and
delimiter and the strategy could more efficiently enumerate the keys.
This is easy for the transient store if we change to a sorted map but
requires some more work for filesystem since we have to issue readdir
selectively.

Could you open a JIRA issue for this and perhaps submit a pull request
to fix it?

On Sun, Dec 03, 2017 at 04:47:38PM +0100, GARDAIS Ionel wrote:
> Hi, 
> 
> I have a question regarding BlobStore listing. 
> We are currently using the FileSystem storage implementation 
> 
> private long containerSize(String containerName) { 
> long containerSize = 0; 
> String marker = null; 
> PageSet<? extends StorageMetadata> containerSML = null; 
> ListContainerOptions lcoRecursive = new ListContainerOptions().recursive(); 
> 
> (1) containerSML = blobStore.list(containerName, lcoRecursive); 
> containerSize += pageSize(containerSML); 
> marker = containerSML.getNextMarker(); 
> 
> while (marker != null) { 
> lcoRecursive.afterMarker(marker); 
> (2) containerSML = blobStore.list(containerName, lcoRecursive); 
> containerSize += pageSize(containerSML); 
> marker = containerSML.getNextMarker(); 
> } 
> 
> log.debug("container:{} size:{}", containerName, containerSize); 
> return containerSize; 
> } 
> private long pageSize(PageSet<? extends StorageMetadata> containerStorageMetadataList) { 
> long size = 0; 
> for (StorageMetadata containerStorageMetadata : containerStorageMetadataList) { 
> if (containerStorageMetadata.getType() == StorageType.BLOB) { 
> log.debug("name:{} size:{}", containerStorageMetadata.getName(), containerStorageMetadata.getSize()); 
> size += containerStorageMetadata.getSize(); 
> } else { 
> log.debug("other in container: {}", containerStorageMetadata.getType()); 
> } 
> } 
> return size; 
> } 
> 
> 
> By calling list() with ListContainerOptions().recursive() on both (1) and (2), all container’s blobs are opened every PageSet. 
> I tried to call list() with recursive() on (1) only then iterating over PageSets without the recursive flag set but I don’t get all Blobs. 
> 
> Is there a way to iterate over all blobs in a container in a more effective way ? 
> 
> Thanks, 
> Ionel 
> 
> --
> 
> 232 avenue Napoleon BONAPARTE 92500 RUEIL MALMAISON
> 
> Capital EUR 219 300,00 - RCS Nanterre B 408 832 301 - TVA FR 09 408 832 301
> 

> BEGIN:VCARD
> VERSION:3.0
> FN:GARDAIS\, Ionel
> N:GARDAIS;Ionel;;;
> ADR;TYPE=work,postal,parcel:;;232 avenue Napol??on BONAPARTE;RUEIL MALMAISON;IdF;92500;FRANCE
> TEL;TYPE=work,voice:0147088131
> EMAIL;TYPE=internet:ionel.gardais@tech-advantage.com
> URL;TYPE=work:http://www.techad.fr
> ORG:TECH advantage
> TITLE:CIO
> REV:2017-09-28T12:56:01Z
> UID:5a4525af-5d0c-4a32-a77b-c565580b116e:114277
> END:VCARD


-- 
Andrew Gaul
http://gaul.org/