You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by to...@apache.org on 2020/07/11 16:15:56 UTC
[libcloud] branch trunk updated (44d59c6 -> f122600)
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git.
from 44d59c6 Update tox target.
new 3d16e9e Update docs and location from which we retrieve pricing data.
new a8ba2f7 Add changelog entry.
new f122600 Update docs.
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.rst | 13 ++++++++++++
docs/compute/pricing.rst | 51 ++++++++++++++++++++++++++++++++++++++++++++++++
libcloud/pricing.py | 8 +++++---
3 files changed, 69 insertions(+), 3 deletions(-)
[libcloud] 01/03: Update docs and location from which we retrieve
pricing data.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 3d16e9e1ed24bbdd4984a5fd53cbda8f261e036a
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Jul 11 18:03:42 2020 +0200
Update docs and location from which we retrieve pricing data.
---
docs/compute/pricing.rst | 44 ++++++++++++++++++++++++++++++++++++++++++++
libcloud/pricing.py | 8 +++++---
2 files changed, 49 insertions(+), 3 deletions(-)
diff --git a/docs/compute/pricing.rst b/docs/compute/pricing.rst
index a01b879..1511dd3 100644
--- a/docs/compute/pricing.rst
+++ b/docs/compute/pricing.rst
@@ -37,6 +37,50 @@ JSON file (``data/pricing.json``) which is bundled with each release. This
pricing data is only updated once you install a new release which means it
could be out of date.
+Downloading latest pricing data from an S3 Bucket
+-------------------------------------------------
+
+Since July 2020, we now run a daily job as part of our CI/CD system which
+scrapes pricing data for various providers and publishes pricing data to a
+public read-only S3 bucket.
+
+Pricing file data is available at the following locations:
+
+* https://libcloud-pricing-data.s3.amazonaws.com/pricing.json
+* https://libcloud-pricing-data.s3.amazonaws.com/pricing.json.sha256
+* https://libcloud-pricing-data.s3.amazonaws.com/pricing.json.sha512
+
+First file contains actual pricing JSON file and the second and third contain
+SHA 256 and SHA 512 sum of that file content.
+
+You can use the content of the sha sum files to implement efficient file
+downloads and only download pricing.json file if the content has changed.
+
+You can do that by fetching the sha sum file, caching the sha sum and only
+downloading ``pricing.json`` file is the sha sum value has changed.
+
+An alternative to using the content of the sha sum file is caching the value
+of the ``ETag`` HTTP response header which you can retrieve by issuing HTTP
+``HEAD`` request against the ``pricing.json`` URL. HEAD request will only
+return the object metdata without the actual content.
+
+For example:
+
+.. sourcecode:: bash
+
+ curl --head https://libcloud-pricing-data.s3.amazonaws.com/pricing.json
+
+ HTTP/1.1 200 OK
+ x-amz-id-2: c8Mer3VtRYWGeKtKlbgwebn3BsVQt+Z/WKKPjk3NcsRSK23BzE6OQDIogzIR2oJGJRmOtS4ydjA=
+ x-amz-request-id: 9A790A3B3587478D
+ Date: Sat, 11 Jul 2020 16:01:39 GMT
+ Last-Modified: Sat, 11 Jul 2020 15:55:50 GMT
+ ETag: "e46324663d76dedafc7d9b09537b18a7"
+ Accept-Ranges: bytes
+ Content-Type: application/json
+ Content-Length: 549390
+ Server: AmazonS3
+
.. _using-custom-pricing-file:
Using a custom pricing file
diff --git a/libcloud/pricing.py b/libcloud/pricing.py
index 695c1c6..70f1fb9 100644
--- a/libcloud/pricing.py
+++ b/libcloud/pricing.py
@@ -43,8 +43,10 @@ __all__ = [
'download_pricing_file'
]
-# Default URL to the pricing file
-DEFAULT_FILE_URL = 'https://git-wip-us.apache.org/repos/asf?p=libcloud.git;a=blob_plain;f=libcloud/data/pricing.json' # NOQA
+# Default URL to the pricing file in a git repo
+DEFAULT_FILE_URL_GIT = 'https://git-wip-us.apache.org/repos/asf?p=libcloud.git;a=blob_plain;f=libcloud/data/pricing.json' # NOQA
+
+DEFAULT_FILE_URL_S3_BUCKET = 'https://libcloud-pricing-data.s3.amazonaws.com/pricing.json' # NOQA
CURRENT_DIRECTORY = os.path.dirname(os.path.abspath(__file__))
DEFAULT_PRICING_FILE_PATH = pjoin(CURRENT_DIRECTORY, 'data/pricing.json')
@@ -190,7 +192,7 @@ def invalidate_module_pricing_cache(driver_type, driver_name):
del PRICING_DATA[driver_type][driver_name]
-def download_pricing_file(file_url=DEFAULT_FILE_URL,
+def download_pricing_file(file_url=DEFAULT_FILE_URL_S3_BUCKET,
file_path=CUSTOM_PRICING_FILE_PATH):
"""
Download pricing file from the file_url and save it to file_path.
[libcloud] 03/03: Update docs.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit f122600d2adf181a9b100cdd552cd02979c5b1b9
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Jul 11 18:15:47 2020 +0200
Update docs.
---
docs/compute/pricing.rst | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
diff --git a/docs/compute/pricing.rst b/docs/compute/pricing.rst
index 1511dd3..b9d7014 100644
--- a/docs/compute/pricing.rst
+++ b/docs/compute/pricing.rst
@@ -50,8 +50,15 @@ Pricing file data is available at the following locations:
* https://libcloud-pricing-data.s3.amazonaws.com/pricing.json.sha256
* https://libcloud-pricing-data.s3.amazonaws.com/pricing.json.sha512
-First file contains actual pricing JSON file and the second and third contain
-SHA 256 and SHA 512 sum of that file content.
+First file contains the actual pricing JSON file and the second and third
+contain SHA 256 and SHA 512 sum of that file content.
+
+We are providing this service free of charge so it's important that you don't
+abuse it. This means you should not download this file more than once per day
+(it makes no sense to do it more often, since it only gets updated once per
+day if there are any changes) and you should utilize one of the caching
+approaches described below and only download ``pricing.json`` file when there
+are any changes / updates.
You can use the content of the sha sum files to implement efficient file
downloads and only download pricing.json file if the content has changed.
@@ -62,7 +69,7 @@ downloading ``pricing.json`` file is the sha sum value has changed.
An alternative to using the content of the sha sum file is caching the value
of the ``ETag`` HTTP response header which you can retrieve by issuing HTTP
``HEAD`` request against the ``pricing.json`` URL. HEAD request will only
-return the object metdata without the actual content.
+return the object metadata without the actual content.
For example:
[libcloud] 02/03: Add changelog entry.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit a8ba2f70a8a8e1eaee9883b8035901e2f1cd9465
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Jul 11 18:12:14 2020 +0200
Add changelog entry.
---
CHANGES.rst | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/CHANGES.rst b/CHANGES.rst
index f56a5d0..c6003dd 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -4,6 +4,19 @@
Changes in Apache Libcloud in development
-----------------------------------------
+Common
+~~~~~~
+
+- ``libcloud.pricing.download_pricing_file`` function has been updated so it
+ tries to download latest ``pricing.json`` file from our public read-only S3
+ bucket.
+
+ We now run a daily job as part of our CI/CD which scrapes provider prices and
+ publishes the latest version of the ``pricing.json`` file to that bucket.
+
+ For more information, please see
+ https://libcloud.readthedocs.io/en/latest/compute/pricing.html.
+
Compute
~~~~~~~