[GitHub] libcloud pull request #1132: Ensure RawResponse downloads don't consume RAM

[pquentin <gi...@git.apache.org>]

GitHub user pquentin opened a pull request:

    https://github.com/apache/libcloud/pull/1132

    Ensure RawResponse downloads don't consume RAM

    
    ## Ensure RawResponse downloads don't consume RAM
    
    The fix is general but the cause is very specific: since libcloud 2.0, `download_object()` in the S3 and GCS drivers first places the item in RAM, then writes it to a file. The main reason it broke is that accessing `body` in a requests Response consumes it entirely and loads it in RAM. To fix this, I moved `body` to a property: it is still accessibly, but loaded only if requested.
    
    Tested this change by downloading a 38M file using this code:
    
    ```
    from libcloud.storage.drivers.google_storage import GoogleStorageDriver
    
    driver = GoogleStorageDriver(key=..., secret=...)
    obj = driver.get_object('container_name', 'object_name')
    driver.download_object(obj, '/path/to/file')
    ```
    
    The RAM usage before:
    
    ![before](https://user-images.githubusercontent.com/42327/31609057-b172a3f0-b283-11e7-9108-2964978bbcd2.png)
    
    The RAM usage after:
    
    ![after](https://user-images.githubusercontent.com/42327/31609063-b846db1a-b283-11e7-97ed-1e6ce365a41b.png)
    
    
    ### Status
    
    - done, ready for review
    
    ### Checklist (tick everything that applies)
    
    - [x] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide) (required, can be done after the PR checks)
    - [ ] Documentation
    - [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/pquentin/libcloud streaming-s3

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/libcloud/pull/1132.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1132
    
----
commit 374cb88c64e4d77c338a9d1b0269d8c01ffc15b8
Author: Quentin Pradet <qu...@clustree.com>
Date:   2017-10-16T10:58:45Z

    Ensure RawResponse downloads don't consume RAM
    
    The fix is general but the cause is very specific: since libcloud 2.0,
    `download_object()` in the S3 and GCS drivers first places the item in
    RAM, then writes it to a file. The reason it broke is that accessing
    `body` in a requests Response consumes it entirely and loads it in RAM.
    To fix this, I moved `body` to a property: it is still accessibly, but
    loaded only if requested.

----


---

[GitHub] libcloud pull request #1132: Ensure RawResponse downloads don't consume RAM

[asfgit <gi...@git.apache.org>]

Github user asfgit closed the pull request at:

    https://github.com/apache/libcloud/pull/1132


---