You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by to...@apache.org on 2018/01/07 18:37:32 UTC
[1/3] libcloud git commit: Ensure RawResponse downloads don't consume
RAM
Repository: libcloud
Updated Branches:
refs/heads/trunk f70264647 -> 4732d83be
Ensure RawResponse downloads don't consume RAM
The fix is general but the cause is very specific: since libcloud 2.0,
`download_object()` in the S3 and GCS drivers first places the item in
RAM, then writes it to a file. The reason it broke is that accessing
`body` in a requests Response consumes it entirely and loads it in RAM.
To fix this, I moved `body` to a property: it is still accessibly, but
loaded only if requested.
Closes #1132
Signed-off-by: Tomaz Muraus <to...@tomaz.me>
Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/31411884
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/31411884
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/31411884
Branch: refs/heads/trunk
Commit: 31411884cd2f989421e378f0bc8fc0c7b6c4783f
Parents: f702646
Author: Quentin Pradet <qu...@clustree.com>
Authored: Mon Oct 16 14:58:45 2017 +0400
Committer: Tomaz Muraus <to...@tomaz.me>
Committed: Sun Jan 7 18:16:32 2018 +0100
----------------------------------------------------------------------
libcloud/common/base.py | 6 +++++-
libcloud/http.py | 4 ++++
2 files changed, 9 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/common/base.py
----------------------------------------------------------------------
diff --git a/libcloud/common/base.py b/libcloud/common/base.py
index 24fdbdb..6dc0589 100644
--- a/libcloud/common/base.py
+++ b/libcloud/common/base.py
@@ -274,12 +274,15 @@ class RawResponse(Response):
if not self._response:
response = self.connection.connection.getresponse()
self._response = HttpLibResponseProxy(response)
- self.body = response.content
if not self.success():
self.parse_error()
return self._response
@property
+ def body(self):
+ return self.response.body
+
+ @property
def reason(self):
if not self._reason:
self._reason = self.response.reason
@@ -581,6 +584,7 @@ class Connection(object):
url=url,
body=data,
headers=headers,
+ raw=raw,
stream=stream)
else:
if retry_enabled:
http://git-wip-us.apache.org/repos/asf/libcloud/blob/31411884/libcloud/http.py
----------------------------------------------------------------------
diff --git a/libcloud/http.py b/libcloud/http.py
index 9149995..5e85026 100644
--- a/libcloud/http.py
+++ b/libcloud/http.py
@@ -319,3 +319,7 @@ class HttpLibResponseProxy(object):
def version(self):
# requests doesn't expose this
return '11'
+
+ @property
+ def body(self):
+ return self._response.content
[2/3] libcloud git commit: Add a comment on why we need to use a
property.
Posted by to...@apache.org.
Add a comment on why we need to use a property.
Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/e9c7e7ef
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/e9c7e7ef
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/e9c7e7ef
Branch: refs/heads/trunk
Commit: e9c7e7ef10aab669bb89e5940dba2f69a85bfd1b
Parents: 3141188
Author: Tomaz Muraus <to...@tomaz.me>
Authored: Sun Jan 7 18:19:33 2018 +0100
Committer: Tomaz Muraus <to...@tomaz.me>
Committed: Sun Jan 7 18:20:24 2018 +0100
----------------------------------------------------------------------
libcloud/common/base.py | 2 ++
libcloud/http.py | 2 ++
2 files changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/libcloud/blob/e9c7e7ef/libcloud/common/base.py
----------------------------------------------------------------------
diff --git a/libcloud/common/base.py b/libcloud/common/base.py
index 6dc0589..17ca286 100644
--- a/libcloud/common/base.py
+++ b/libcloud/common/base.py
@@ -280,6 +280,8 @@ class RawResponse(Response):
@property
def body(self):
+ # Note: We use property to avoid saving whole response body into RAM
+ # See https://github.com/apache/libcloud/pull/1132 for details
return self.response.body
@property
http://git-wip-us.apache.org/repos/asf/libcloud/blob/e9c7e7ef/libcloud/http.py
----------------------------------------------------------------------
diff --git a/libcloud/http.py b/libcloud/http.py
index 5e85026..e691937 100644
--- a/libcloud/http.py
+++ b/libcloud/http.py
@@ -322,4 +322,6 @@ class HttpLibResponseProxy(object):
@property
def body(self):
+ # NOTE: We use property to avoid saving whole response body into RAM
+ # See https://github.com/apache/libcloud/pull/1132 for details
return self._response.content
[3/3] libcloud git commit: Add a changelog entry for #1132.
Posted by to...@apache.org.
Add a changelog entry for #1132.
Project: http://git-wip-us.apache.org/repos/asf/libcloud/repo
Commit: http://git-wip-us.apache.org/repos/asf/libcloud/commit/4732d83b
Tree: http://git-wip-us.apache.org/repos/asf/libcloud/tree/4732d83b
Diff: http://git-wip-us.apache.org/repos/asf/libcloud/diff/4732d83b
Branch: refs/heads/trunk
Commit: 4732d83be96dfa19422f741fccd8ff59e27d19ff
Parents: e9c7e7e
Author: Tomaz Muraus <to...@tomaz.me>
Authored: Sun Jan 7 19:28:54 2018 +0100
Committer: Tomaz Muraus <to...@tomaz.me>
Committed: Sun Jan 7 19:28:54 2018 +0100
----------------------------------------------------------------------
CHANGES.rst | 8 ++++++++
1 file changed, 8 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/libcloud/blob/4732d83b/CHANGES.rst
----------------------------------------------------------------------
diff --git a/CHANGES.rst b/CHANGES.rst
index c31f54b..f402a56 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -126,6 +126,14 @@ Storage
- [Digital Ocean Spaces] Add support for AMS3 region (GITHUB-1142)
[Andrew Starr-Bochicchio]
+- Fix a bug / regression which resulted in increased memory consumption when
+ using ``download_object`` method. This method would store whole object
+ content in memory even though there was no need for that.
+
+ This regression was introduced in 2.0.0 when we moved to using ``requests``
+ library.
+ (GITHUB-1132)
+ [Quentin Pradet]
Changes in Apache Libcloud 2.2.1
--------------------------------