libcloud connection management

["Sullivan, B" <B....@nttdata.com>]

I'm a new user of Apache libcloud. I'm currently using the Amazon S3 storage driver with Python 2.7.  I'm basically doing two libcloud calls in my program:

- getObject : To determine if the object name is already being used

- uploadObject : To perform the actual file upload



When I set the root logger to DEBUG I'm able to see the following:

20181111165749.872 Thread-2    _new_conn   INFO: Starting new HTTPS connection (2): s3-us-east-2.amazonaws.com

20181111165750.254 Thread-2   _make_request  DEBUG: "HEAD /mycontainer HTTP/1.1" 200 0

20181111165750.255 Thread-2    _new_conn   INFO: Starting new HTTPS connection (3): s3-us-east-2.amazonaws.com

20181111165750.638 Thread-2   _make_request  DEBUG: "HEAD /mycontainer/myobject HTTP/1.1" 404 0

20181111165750.640 Thread-2    _new_conn   INFO: Starting new HTTPS connection (4): s3-us-east-2.amazonaws.com

20181111165751.415 Thread-2   _make_request  DEBUG: "PUT /mycontainer/myobject HTTP/1.1" 200 0



I attribute the first two HEAD commands to the getObject() call and the PUT is obviously from the uploadObject() call. What I'm curious about is why each command is prefaced by the call to _new_conn().  Is there a way for the library to reuse connections to make the program more efficient?



Thanks for the assistance.



Brian



Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged,confidential, and proprietary data. If you are not the intended recipient,please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.

Re: libcloud connection management

[Quentin Pradet <qu...@gmail.com>]

Hello,

libcloud uses requests, which uses urllib3 for connection pooling. The logs
you are seeing are from urllib3. I suggest you try logging the calls that
libcloud makes to requests in libcloud.http (
https://github.com/apache/libcloud/blob/trunk/libcloud/http.py#L209-L222).
This will help to see if the bug is in libcloud or not.

Hope that helps,
Quentin