You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2022/06/20 10:19:04 UTC
[GitHub] [libcloud] Eis-D-Z opened a new pull request, #1715: Use official endpoint for EC2 prices
Eis-D-Z opened a new pull request, #1715:
URL: https://github.com/apache/libcloud/pull/1715
## EC2 prices scraping script upgrade
### Use official endpoint to get EC2 prices to get as many prices and as acurate as possible. This enables to add prices for instances with suse, rhel and rhel with high availability.
This like the previous script targets only OnDemand instance prices
### Status
- done, ready for review
### Checklist (tick everything that applies)
- [x] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide) (required, can be done after the PR checks)
- [x] Documentation
- [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
- [x] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes) (required for bigger changes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1240567745
I added a note on pricing file size + potential small memory usage increase (ce5b74479f4d21f2928d2d2b56d1cf734c4ee38a) and also updated the script to include progress bar when parsing sku pricing data (7bffd095b151d451bf6f723e89a88745c52399a8).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Eis-D-Z commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1237324605
@Kami Ready, sorry for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Eis-D-Z commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r912785714
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
Review Comment:
Yes, this code was taken from the size scraping file. The only good thing is that it is enough to be downloaded once for both scripts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Eis-D-Z commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r912810371
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ return download_json()
+
+
+# Prices and sizes are in different dicts and categorized by sku
+def get_all_prices():
+ # return variable
+ # prices = {sku : {price: int, unit: string}}
+ prices = {}
+ current_sku = ""
+ current_rate_code = ""
+ amazonEC2_offer_code = "JRTCKXETXF"
Review Comment:
The file has two big keys, "products" and "terms". Under "products" are the technical details of each sku.
Under "terms" there are pricing data. A sample entry is like this:
{ "terms": { "onDemand": { sku: { sku: {"JRTCKXETXF": { "priceDimensions": { rateCode: {"unit": "$", "value": "0.17" } } } } } } }
It is fortunate that this ijson parser has been used since it makes it easy to find the "rateCode" number (with ijson you parse every line and it tells you if what you encounter is a key or a value). Otherwise it would be more difficult. I guess that amazon has a way of determining the rateCode based on the sku or some other data but I haven't found their relation.
The JRTCKXETXF is the same for all EC2 items and I named it amazonEC2_offer_code.
Thus what the code does is parse the "terms" dict line by like, when it encounters a dict key under terms["OnDemand"] set it as the current_sku. Then search the next lines for the rateCode of the item and after that finally get the price and the unit of the sku. Save those two in the dict to be returned under the sku. ({sku1: {"price": "xx", "unit": "$"}, sku2:{... }
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1240567842
Merged, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1169181135
I've added some inline comments / questions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1193116027
I've pushed some small lint and ``tox.ini`` fixes, but it looks like the script still fails with empty cache:
```
Traceback (most recent call last):
File "contrib/scrape-ec2-prices.py", line 301, in <module>
main()
File "contrib/scrape-ec2-prices.py", line 294, in main
pricing_data = scrape_ec2_pricing()
File "contrib/scrape-ec2-prices.py", line 137, in scrape_ec2_pricing
prices = get_all_prices()
File "contrib/scrape-ec2-prices.py", line 104, in get_all_prices
with open(json_file, 'r') as f:
TypeError: expected str, bytes or os.PathLike object, not tuple
```
I pushed a small fix for that, but didn't have time to dig in if there are other issues present.
On top of that, it looks like script takes a very long time to complete even when cache is available (I never got it to finish - not sure if it just takes a really long time or there is an infinite loop or similar).
What kind of run times have you seen on your computer?
If it takes a really long time, it might also not be a bad idea to add some kind of basic progress bar (e.g. print "." every now and then to give some feedback that script didn't get stuck or even better, periodically print how many items we have processed so far out of the total number).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] asfgit merged pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
asfgit merged PR #1715:
URL: https://github.com/apache/libcloud/pull/1715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r908895069
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
Review Comment:
May be a good idea to use with context manager to ensure file is closed (then again, this script is short lived so any file handle leaks are not a big deal)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] codecov-commenter commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1237454611
# [Codecov](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#1715](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f44038b) into [trunk](https://codecov.io/gh/apache/libcloud/commit/77967bfaf672a68b65be56c0fdbae2e280d1e02d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (77967bf) will **not change** coverage.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## trunk #1715 +/- ##
=======================================
Coverage 83.28% 83.28%
=======================================
Files 400 400
Lines 87862 87862
Branches 10708 10708
=======================================
Hits 73180 73180
Misses 11495 11495
Partials 3187 3187
```
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [77967bf...f44038b](https://codecov.io/gh/apache/libcloud/pull/1715?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1240545310
I had a look and by default we only cache data in memory for drivers for which pricing data is requested by the using so this will only affect people using EC2 driver.
I will still add a note about that to the changelog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1193116601
EDIT: It looks like ``ijson.parse()`` takes a very long time already and then iteration over the parsed content.
I don't think it's possible to add progress bar for ``ijson.parse()`` step itself (we would need to modify ijson or similar), but we might be able to add it for actual code where we process parsed response.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1236669545
@Eis-D-Z How is this PR looking? Would be great if we could get it wrapped up merged. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r908895328
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ return download_json()
+
+
+# Prices and sizes are in different dicts and categorized by sku
+def get_all_prices():
+ # return variable
+ # prices = {sku : {price: int, unit: string}}
+ prices = {}
+ current_sku = ""
+ current_rate_code = ""
+ amazonEC2_offer_code = "JRTCKXETXF"
Review Comment:
What is this code used for? Just trying to understand how it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1169177801
Great, thanks!
Just curious is this official pricing endpoint any smaller? Or it's perhaps even bigger? Just curious since previous JSON we used to download was massive (multi GB) which meant that download could take a while (but we already downloaded it in chunks to avoid excessive memory usage + added progress bar).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Eis-D-Z commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r912785714
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
Review Comment:
Yes, this code was taken from the size scraping file, I will update to the latest version though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1237456266
@Eis-D-Z Thanks! Since this change basically doubled the pricing file (from 2 MB to 3 MB), I will take some time to verify that it has no negative implications on memory usage and similar (we had similar issues with sizing file in the past and to avoid those issues, we split some of the stuff into different files and made some imports lazy and only exercised in case EC2 driver is used).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r908894718
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
Review Comment:
I assume this requires us to cache whole massive JSON dictionary in memory, right?
I guess it's fine and we probably already did the same before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1193141856
@Eis-D-Z Script seems to be working now and other checks passing, but unit tests one is still failing - https://github.com/apache/libcloud/runs/7481787304?check_suite_focus=true
It appears it's failing because the pricing.json file format has changed. Or is something else going on? Can you please have a look when you get a chance? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Kami commented on pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Kami commented on PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#issuecomment-1193121557
I pushed some additional changes to increase buffer size to speed up parsing a bit and to print up a progress bar - since ``ijson`` handles parsing in an incremental manner and returns a generator there is no easy way for us to determine how many more items we need to parse / how far along we are so we can only show that parsing is being performed, but we can't display how many items there are left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [libcloud] Eis-D-Z commented on a diff in pull request #1715: Use official endpoint for EC2 prices
Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on code in PR #1715:
URL: https://github.com/apache/libcloud/pull/1715#discussion_r912810600
##########
contrib/scrape-ec2-prices.py:
##########
@@ -161,82 +48,155 @@
"extra-large",
]
-RE_NUMERIC_OTHER = re.compile(r"(?:([0-9]+)|([-A-Z_a-z]+)|([^-0-9A-Z_a-z]+))")
-
-BASE_PATH = os.path.dirname(os.path.abspath(__file__))
-PRICING_FILE_PATH = os.path.join(BASE_PATH, "../libcloud/data/pricing.json")
-PRICING_FILE_PATH = os.path.abspath(PRICING_FILE_PATH)
-
+def download_json():
+ response = requests.get(URL, stream=True)
+ try:
+ return open(TEMPFILE, "r")
+ except IOError:
+ with open(TEMPFILE, "wb") as fo:
+ for chunk in response.iter_content(chunk_size=2**20):
+ if chunk:
+ fo.write(chunk)
+ return open(TEMPFILE, "r")
+
+
+def get_json():
+ try:
+ return open(TEMPFILE, "r")
Review Comment:
Right. I will fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@libcloud.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org