You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by GitBox <gi...@apache.org> on 2020/06/17 11:52:23 UTC

[GitHub] [libcloud] Eis-D-Z opened a new pull request #1469: Pricing aws

Eis-D-Z opened a new pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469


   ## Changes Title 
   EC2 pricing update
   
   ### Description
   
   The main url from which prices were scraped is deprecated, this introduces a new url and a  new location (Stockholm). The format in which the prices are to be saved in the pricing.json file is also changed, now it is ec2_linux (or ec2_windows) --> size --> region --> price. This was done for convenience, some regions do not have some sizes and in other clouds the price might default to another region, this way it is easier to introduce a default, also it is easier to add prices for different premium images in the future like RHEL and SLES. For the moment only windows prices are being scraped besides the linux ones (free images), even though for libcloud's current needs windows prices are not necessary. The driver is also changed so prices can be fetched correctly. This request does not include the resulting constants.py and pricing.json after the scripts are run. If it should please let me know.
   
   ### Status
   - done, ready for review
   
   ### Checklist (tick everything that applies)
   
   - [x] [Code linting](http://libcloud.readthedocs.org/en/latest/development.html#code-style-guide) (required, can be done after the PR checks)
   - [x] Documentation
   - [ ] [Tests](http://libcloud.readthedocs.org/en/latest/testing.html)
   - [x] [ICLA](http://libcloud.readthedocs.org/en/latest/development.html#contributing-bigger-changes) (required for bigger changes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Eis-D-Z commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653554679


   @Kami I believe that the first option you proposed is the best for the moment. That is adding a region argument like `def get_size_price(driver_type, driver_name, size_id, region=None)` and later in the code check if a region was given and get the price based on the region  `price = float(pricing[size_id][region])`, and if region is None then as it was until now.
   
   In our day to day, cost and pricing is a hot demand thus I kinda foresee that this first step in this direction has its good side, I expect that eventually an all around and standard format will be reached, until then I would treat this just as a first step in that direction. 
   
   If you can confirm that the first option you proposed is acceptable I will proceed with the necessary changes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] asfgit merged pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
asfgit merged pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653051232


   I fixed the sorting locally, but there is another error in how we retrieve pricing data inside the driver code.
   
   ```bash
   Traceback (most recent call last):
     File "/home/kami/w/libcloud/libcloud/libcloud/test/compute/__init__.py", line 32, in test_list_sizes_response
       sizes = self.driver.list_sizes()
     File "/home/kami/w/libcloud/libcloud/libcloud/compute/drivers/ec2.py", line 6210, in list_sizes
       size_id=instance_type)
     File "/home/kami/w/libcloud/libcloud/libcloud/pricing.py", line 150, in get_size_price
       price = float(pricing[size_id])
   TypeError: float() argument must be a string or a number, not 'dict'
   ```
   
   I'll also push a fix for that, but I'm also slightly worried about this change - some other driver classes also rely on the base EC2 driver and we need to be sure this PR doesn't introduce inadvertent regressions and brakes other drivers or similar.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on a change in pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on a change in pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#discussion_r453204143



##########
File path: contrib/scrape-ec2-prices.py
##########
@@ -234,15 +267,20 @@ def sort_key_by_numeric_other(key_value):
     """
     Split key into numeric, alpha and other part and sort accordingly.
     """
-    return tuple((
-        int(numeric) if numeric else None,
-        INSTANCE_SIZES.index(alpha) if alpha in INSTANCE_SIZES else alpha,
-        other
-    ) for (numeric, alpha, other) in RE_NUMERIC_OTHER.findall(key_value[0]))
+    result = []
+
+    for (numeric, alpha, other) in RE_NUMERIC_OTHER.findall(key_value[0]):
+        numeric = int(numeric) if numeric else -1
+        alpha = INSTANCE_SIZES.index(alpha) if alpha in INSTANCE_SIZES else alpha
+        alpha = str(alpha)
+        item = tuple([numeric, alpha, other])
+        result.append(item)
+
+    return tuple(result)
 
 
 def main():
-    print('Scraping EC2 pricing data')
+    print('Scraping EC2 pricing data (this may take up to 2 minutes)....')
 
     pricing_data = scrape_ec2_pricing()
     update_pricing_file(pricing_file_path=PRICING_FILE_PATH,

Review comment:
       This function is repeated across multiple scripts so eventually we should refactor it in some common place.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-656041884


   @Eis-D-Z Please let me know when this is ready for a review (I see tests are still failing).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653561574


   Let's please just make sure that format change is documented in the release notes, in case someone is directly using data from that file (unlikely, but just in case). 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Eis-D-Z commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Eis-D-Z commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-651753429


   I ran the scripts after all and added the new pricing.json to the commit, otherwise the tests would not pass (they should pass now)


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-652332730


   Thanks for the contribution.
   
   The build is failing, can you please have a  look?
   
   In addition to that, running the script locally returns this error:
   
   ```
   Scraping EC2 pricing data
   Traceback (most recent call last):
     File "contrib/scrape-ec2-prices.py", line 288, in <module>
       main()
     File "contrib/scrape-ec2-prices.py", line 282, in main
       pricing_data=pricing_data)
     File "contrib/scrape-ec2-prices.py", line 240, in update_pricing_file
       data = sort_nested_dict(data)
     File "contrib/scrape-ec2-prices.py", line 259, in sort_nested_dict
       result[key] = sort_nested_dict(value)
     File "contrib/scrape-ec2-prices.py", line 259, in sort_nested_dict
       result[key] = sort_nested_dict(value)
     File "contrib/scrape-ec2-prices.py", line 257, in sort_nested_dict
       for key, value in sorted(value.items(), key=sort_key_by_numeric_other):
   TypeError: '<' not supported between instances of 'int' and 'str'
   ```
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653055900


   @Eis-D-Z ^ what's your take on this?
   
   I'm worried because ``get_size_price`` function expects pricing data in pricing.json file to be in a specific format. And changes in this PR brake promise about that format (aka they change the format, but for EC2 only).
   
   One option is to add support for region argument to that function or perhaps add new ``get_size_price_for_region`` function and make sure we support both pricing formats.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653555747


   Yeah, that sounds good to me. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [libcloud] Kami commented on pull request #1469: Pricing aws

Posted by GitBox <gi...@apache.org>.
Kami commented on pull request #1469:
URL: https://github.com/apache/libcloud/pull/1469#issuecomment-653054245


   After some more digging in, I noticed that issue is related to the change of the data format / schema you did in pricing.json file.
   
   Previously we basically mapped ``<provider constant>_<region>``: ``<instance type>`` -> ``<float price>``.
   
   With your change we now have two additional layers of in-directions and we map Instance type (e.g. ec2 linux) to region to instance types pricing.
   
   In theory I'm fine with those changes (they support more complex pricing scheme) as long as we don't rush it and make sure they don't brake any of the existing code or drivers.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org