[GitHub] [libcloud] Ido-Levi opened a new issue #1583: container.list_objects with prefix and the local driver is very slow when a lot of files are stored

[GitBox <gi...@apache.org>]

Ido-Levi opened a new issue #1583:
URL: https://github.com/apache/libcloud/issues/1583


   ## Summary
   
   I have a large storage that I'm using libcloud to manage - in this specific case it's around 500,000 objects, and I've noticed that listing object using the local driver and a prefix is very slow (took almost 2 minutes to return 90 objects).
   
   ## Detailed Information
   In the local driver implementation when calling list_objects with a prefix - we firstly get all the objects and serialize them and just after that we are filtering by the prefix.
   
   Found on python3.6.8 with libcloud 3.2.0 and ubuntu 16:04
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [libcloud] Ido-Levi commented on issue #1583: container.list_objects with prefix and the local driver is very slow when a lot of files are stored

[GitBox <gi...@apache.org>]

Ido-Levi commented on issue #1583:
URL: https://github.com/apache/libcloud/issues/1583#issuecomment-847873196


   PR: https://github.com/apache/libcloud/pull/1584


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org