You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by to...@apache.org on 2023/07/31 15:05:32 UTC

[libcloud] branch trunk updated (aa9a947bb -> 8d0de1093)

This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git


    from aa9a947bb Add changelog entry for #1883 and #1884.
     add ad5bd6884 fix list_nodes for the azure arm driver
     add 579f625fd add unit tests
     add b8e836c07 Merge branch 'trunk' into fix-azure-list-nodes
     new c9060092b Merge branch 'fix-azure-list-nodes' of https://github.com/jan-mue/libcloud into jan-mue-fix-azure-list-nodes
     new c49ecd0a4 Update the code so we don't end up in an infinite loop in case API returns invalid response or similar.
     new cc3085950 Bump the pagination limit to 300s.
     new 09cc6c449 Add changelog entry for #1824 #1850.
     new efe4978f6 Update the comment.
     new b843334e4 Fix codecov config.
     new 863b88df1 Merge branch 'trunk' into fix-azure-list-nodes
     new 8d0de1093 Add comment which clarifies number of API requests performed by the method.

The 8 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .codecov.yml                                       |  4 ---
 CHANGES.rst                                        |  5 +++
 libcloud/compute/drivers/azure_arm.py              | 37 ++++++++++++++++++----
 ...777_oauth2_token_PAGINATION_INFINITE_LOOP.json} |  0
 ...roviders_Microsoft_Compute_virtualMachines.json |  3 +-
 ...iders_Microsoft_Compute_virtualMachines_1.json} | 12 +++----
 ..._virtualMachines_PAGINATION_INFINITE_LOOP.json} |  9 ++----
 ..._virtualMachines_test_node_2_InstanceView.json} |  0
 ...Network_networkInterfaces_test_node_2_nic.json} | 16 +++++-----
 libcloud/test/compute/test_azure_arm.py            | 34 ++++++++++++++++++--
 10 files changed, 85 insertions(+), 35 deletions(-)
 copy libcloud/test/compute/fixtures/azure_arm/{_77777777_7777_7777_7777_777777777777_oauth2_token.json => _77777777_7777_7777_7777_777777777777_oauth2_token_PAGINATION_INFINITE_LOOP.json} (100%)
 copy libcloud/test/compute/fixtures/azure_arm/{_subscriptions_99999999_providers_Microsoft_Compute_virtualMachines.json => _subscriptions_99999999_providers_Microsoft_Compute_virtualMachines_1.json} (89%)
 copy libcloud/test/compute/fixtures/azure_arm/{_subscriptions_99999999_providers_Microsoft_Compute_virtualMachines.json => _subscriptions_99999999_providers_Microsoft_Compute_virtualMachines_PAGINATION_INFINITE_LOOP.json} (82%)
 copy libcloud/test/compute/fixtures/azure_arm/{_subscriptions_99999999_resourceGroups_000000_providers_Microsoft_Compute_virtualMachines_test_node_1_InstanceView.json => _subscriptions_99999999_resourceGroups_000000_providers_Microsoft_Compute_virtualMachines_test_node_2_InstanceView.json} (100%)
 copy libcloud/test/compute/fixtures/azure_arm/{_subscriptions_99999999_resourceGroups_000000_providers_Microsoft_Network_networkInterfaces_test_node_1_nic.json => _subscriptions_99999999_resourceGroups_000000_providers_Microsoft_Network_networkInterfaces_test_node_2_nic.json} (77%)


[libcloud] 06/08: Fix codecov config.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit b843334e4ac13e44126bb2ccf2fd113efb37c9c9
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 16:57:01 2023 +0200

    Fix codecov config.
---
 .codecov.yml | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/.codecov.yml b/.codecov.yml
index 925a7a72e..61caf6f46 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -1,7 +1,3 @@
-codecov:
-  #notify:
-    #require_ci_to_pass: yes
-
 coverage:
   precision: 2         # decimal places to display: 0 <= value <= 4
   round: nearest


[libcloud] 07/08: Merge branch 'trunk' into fix-azure-list-nodes

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit 863b88df1f1a8fad42279d5fee38ec9893427b77
Merge: efe4978f6 b843334e4
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 16:57:12 2023 +0200

    Merge branch 'trunk' into fix-azure-list-nodes

 .codecov.yml | 4 ----
 1 file changed, 4 deletions(-)


[libcloud] 08/08: Add comment which clarifies number of API requests performed by the method.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit 8d0de10937321bf4099c9970cd683fa8fbecb572
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 17:00:54 2023 +0200

    Add comment which clarifies number of API requests performed by the
    method.
---
 libcloud/compute/drivers/azure_arm.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libcloud/compute/drivers/azure_arm.py b/libcloud/compute/drivers/azure_arm.py
index 1a7ee2327..224b831b1 100644
--- a/libcloud/compute/drivers/azure_arm.py
+++ b/libcloud/compute/drivers/azure_arm.py
@@ -463,8 +463,10 @@ class AzureNodeDriver(NodeDriver):
 
         :return:  list of node objects
         :rtype: ``list`` of :class:`.Node`
-        """
 
+        NOTE: With the default arguments, the function may result in M * (1 + (N * 3)) HTTP
+        requests where M is number of API pages and N is number of nodes returned per page.
+        """
         if ex_resource_group:
             action = (
                 "/subscriptions/%s/resourceGroups/%s/"


[libcloud] 01/08: Merge branch 'fix-azure-list-nodes' of https://github.com/jan-mue/libcloud into jan-mue-fix-azure-list-nodes

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit c9060092b793141bb10c74ce2053f8b71c98db3b
Merge: aa9a947bb b8e836c07
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 15:21:50 2023 +0200

    Merge branch 'fix-azure-list-nodes' of https://github.com/jan-mue/libcloud into jan-mue-fix-azure-list-nodes

 libcloud/compute/drivers/azure_arm.py              | 18 ++++++----
 ...roviders_Microsoft_Compute_virtualMachines.json |  3 +-
 ...iders_Microsoft_Compute_virtualMachines_1.json} | 12 +++----
 ...e_virtualMachines_test_node_2_InstanceView.json | 27 +++++++++++++++
 ..._Network_networkInterfaces_test_node_2_nic.json | 38 ++++++++++++++++++++++
 libcloud/test/compute/test_azure_arm.py            | 21 ++++++++++--
 6 files changed, 103 insertions(+), 16 deletions(-)



[libcloud] 04/08: Add changelog entry for #1824 #1850.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit 09cc6c4496f066736d216403eb69053d8f42cdfc
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 15:59:21 2023 +0200

    Add changelog entry for #1824 #1850.
---
 CHANGES.rst | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/CHANGES.rst b/CHANGES.rst
index fde5d2fa6..3a0b176f8 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -19,6 +19,11 @@ Compute
   (GITHUB-1877)
   [Matthias Gatto - @outscale-mgo]
 
+- [Azure ARM] Implement pagination in the ``list_nodes()`` method. This fixes a
+  bug which would result in sometimes not all nodes being returned.
+  (GITHUB-1824, GITHUB-1850)
+  [Jan Müller - @ojan-mue]
+
 Storage
 ~~~~~~~
 


[libcloud] 05/08: Update the comment.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit efe4978f6406d79bfefd48f88df892be5172d7af
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 16:02:06 2023 +0200

    Update the comment.
---
 libcloud/compute/drivers/azure_arm.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/libcloud/compute/drivers/azure_arm.py b/libcloud/compute/drivers/azure_arm.py
index 678f6ae6d..1a7ee2327 100644
--- a/libcloud/compute/drivers/azure_arm.py
+++ b/libcloud/compute/drivers/azure_arm.py
@@ -66,6 +66,9 @@ VM_SIZE_API_VERSION = "2015-06-15"  # this API is deprecated
 
 # If pagination code in the list_nodes() method has still not completed after this mount of
 # seconds, we will break early from while True loop to avoid infinite loop under edge conditions.
+# Keep in mind that we want this timeout relatively high since each `_to_node()` method call which
+# is called for each node can result in additional HTTP requests (to retrieve power state, nics,
+# etc).
 LIST_NODES_PAGINATION_TIMEOUT = 300
 
 


[libcloud] 02/08: Update the code so we don't end up in an infinite loop in case API returns invalid response or similar.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit c49ecd0a4b8fe1e3820d8764f9aded21e5dee11c
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 15:46:26 2023 +0200

    Update the code so we don't end up in an infinite loop in case API
    returns invalid response or similar.
    
    Also add a test case for it.
---
 libcloud/compute/drivers/azure_arm.py              | 18 ++++++--
 ...7777_oauth2_token_PAGINATION_INFINITE_LOOP.json |  1 +
 ...e_virtualMachines_PAGINATION_INFINITE_LOOP.json | 53 ++++++++++++++++++++++
 libcloud/test/compute/test_azure_arm.py            | 17 ++++++-
 4 files changed, 84 insertions(+), 5 deletions(-)

diff --git a/libcloud/compute/drivers/azure_arm.py b/libcloud/compute/drivers/azure_arm.py
index 0d853285c..82273c565 100644
--- a/libcloud/compute/drivers/azure_arm.py
+++ b/libcloud/compute/drivers/azure_arm.py
@@ -25,7 +25,7 @@ import base64
 import binascii
 
 from libcloud.utils import iso8601
-from libcloud.utils.py3 import basestring, urlparse, parse_qs
+from libcloud.utils.py3 import parse_qs, urlparse, basestring
 from libcloud.common.types import LibcloudError
 from libcloud.compute.base import (
     Node,
@@ -64,6 +64,10 @@ VM_API_VERSION = "2021-11-01"
 VM_EXTENSION_API_VERSION = "2015-06-15"
 VM_SIZE_API_VERSION = "2015-06-15"  # this API is deprecated
 
+# If pagination code in the list_nodes() method has still not completed after this mount of
+# seconds, we will break early from while True loop to avoid infinite loop under edge conditions.
+LIST_NODES_PAGINATION_TIMEOUT = 60
+
 
 class AzureImage(NodeImage):
     """Represents a Marketplace node image that an Azure VM can boot from."""
@@ -469,11 +473,19 @@ class AzureNodeDriver(NodeDriver):
                 self.subscription_id
             )
         params = {"api-version": VM_API_VERSION}
+
+        now_ts = int(time.time())
+        deadline_ts = now_ts + LIST_NODES_PAGINATION_TIMEOUT
+
         nodes = []
-        while True:
+        while time.time() < deadline_ts:
             r = self.connection.request(action, params=params)
-            nodes.extend(self._to_node(n, fetch_nic=ex_fetch_nic, fetch_power_state=ex_fetch_power_state) for n in r.object["value"])
+            nodes.extend(
+                self._to_node(n, fetch_nic=ex_fetch_nic, fetch_power_state=ex_fetch_power_state)
+                for n in r.object["value"]
+            )
             if not r.object.get("nextLink"):
+                # No next page
                 break
             parsed_next_link = urlparse.urlparse(r.object["nextLink"])
             params.update({k: v[0] for k, v in parse_qs(parsed_next_link.query).items()})
diff --git a/libcloud/test/compute/fixtures/azure_arm/_77777777_7777_7777_7777_777777777777_oauth2_token_PAGINATION_INFINITE_LOOP.json b/libcloud/test/compute/fixtures/azure_arm/_77777777_7777_7777_7777_777777777777_oauth2_token_PAGINATION_INFINITE_LOOP.json
new file mode 100644
index 000000000..b9702f2a8
--- /dev/null
+++ b/libcloud/test/compute/fixtures/azure_arm/_77777777_7777_7777_7777_777777777777_oauth2_token_PAGINATION_INFINITE_LOOP.json
@@ -0,0 +1 @@
+{"expires_in":"3600","token_type":"Bearer","expires_on":"1111111111","not_before":"1111111111","resource":"https://management.core.windows.net/","access_token":"3333333333333333333333333333333333333333333333333333333"}
diff --git a/libcloud/test/compute/fixtures/azure_arm/_subscriptions_99999999_providers_Microsoft_Compute_virtualMachines_PAGINATION_INFINITE_LOOP.json b/libcloud/test/compute/fixtures/azure_arm/_subscriptions_99999999_providers_Microsoft_Compute_virtualMachines_PAGINATION_INFINITE_LOOP.json
new file mode 100644
index 000000000..8c9e9135d
--- /dev/null
+++ b/libcloud/test/compute/fixtures/azure_arm/_subscriptions_99999999_providers_Microsoft_Compute_virtualMachines_PAGINATION_INFINITE_LOOP.json
@@ -0,0 +1,53 @@
+{
+  "value": [
+    {
+      "properties": {
+        "vmId": "CCEEBF63-E92B-4A50-9949-6E44BFC61D3F",
+        "additionalCapabilities": {
+          "ultraSSDEnabled": "False",
+          "hibernationEnabled": "False"
+        },
+        "hardwareProfile": {
+          "vmSize": "Standard_A1"
+        },
+        "storageProfile": {
+          "imageReference": {
+            "publisher": "OpenLogic",
+            "offer": "CentOS",
+            "sku": "7.3",
+            "version": "latest"
+          },
+          "osDisk": {
+            "osType": "Linux",
+            "name": "test-node-disk-1",
+            "createOption": "FromImage",
+            "caching": "ReadWrite",
+            "managedDisk": {
+              "storageAccountType": "Standard_LRS",
+              "id": "/subscriptions/99999999-9999-9999-9999-999999999999/resourceGroups/000000/providers/Microsoft.Compute/disks/test-node-disk-1"
+            }
+          },
+          "dataDisks": []
+        },
+        "osProfile": {
+          "computerName": "test-node-1",
+          "adminUsername": "user",
+          "linuxConfiguration": {
+            "disablePasswordAuthentication": false
+          },
+          "secrets": []
+        },
+        "networkProfile": {
+          "networkInterfaces": []
+        },
+        "provisioningState": "Running"
+      },
+      "type": "Microsoft.Compute/virtualMachines",
+      "location": "eastus",
+      "tags": {},
+      "id": "/subscriptions/99999999-9999-9999-9999-999999999999/resourceGroups/000000/providers/Microsoft.Compute/virtualMachines/test-node-1",
+      "name": "test-node-1"
+    }
+  ],
+  "nextLink": "https://management.azure.com:443/subscriptions/99999999-9999-9999-9999-999999999999/providers/Microsoft.Compute/virtualMachines?api-version=2021-11-01&$skiptoken=1!/Subscriptions/99999999-9999-9999-9999-999999999999/ResourceGroups/000000/VMs/DDFEBF64-E92B-4A50-9949-6E44BFC61D4G"
+}
diff --git a/libcloud/test/compute/test_azure_arm.py b/libcloud/test/compute/test_azure_arm.py
index 0795d0ec2..b81fb24b5 100644
--- a/libcloud/test/compute/test_azure_arm.py
+++ b/libcloud/test/compute/test_azure_arm.py
@@ -20,7 +20,7 @@ from datetime import datetime
 from unittest import mock
 
 from libcloud.test import MockHttp, LibcloudTestCase, unittest
-from libcloud.utils.py3 import httplib, urlparse, parse_qs, urlunquote
+from libcloud.utils.py3 import httplib, parse_qs, urlparse, urlunquote
 from libcloud.common.types import LibcloudError
 from libcloud.compute.base import NodeSize, NodeLocation, StorageVolume, VolumeSnapshot
 from libcloud.compute.types import Provider, NodeState, StorageVolumeState, VolumeSnapshotState
@@ -42,6 +42,7 @@ class AzureNodeDriverTests(LibcloudTestCase):
     APPLICATION_PASS = "p4ssw0rd"
 
     def setUp(self):
+        AzureMockHttp.type = None
         Azure = get_driver(Provider.AZURE_ARM)
         Azure.connectionCls.conn_class = AzureMockHttp
         self.driver = Azure(
@@ -458,6 +459,18 @@ class AzureNodeDriverTests(LibcloudTestCase):
 
         fps_mock.assert_called()
 
+    @mock.patch(
+        "libcloud.compute.drivers.azure_arm.AzureNodeDriver._fetch_power_state",
+        return_value=NodeState.UPDATING,
+    )
+    @mock.patch("libcloud.compute.drivers.azure_arm.LIST_NODES_PAGINATION_TIMEOUT", 1)
+    def test_list_nodes_pagination_timeout_reached(self, fps_mock):
+        # Verify we don't end up in an infinite loop in case server returns a bad response or
+        # similar
+        AzureMockHttp.type = "PAGINATION_INFINITE_LOOP"
+        nodes = self.driver.list_nodes()
+        self.assertTrue(len(nodes) >= 1)
+
     @mock.patch(
         "libcloud.compute.drivers.azure_arm.AzureNodeDriver" "._fetch_power_state",
         return_value=NodeState.UPDATING,
@@ -824,7 +837,7 @@ class AzureMockHttp(MockHttp):
                 AzureNodeDriverTests.SUBSCRIPTION_ID,
             )
             unquoted_url = urlunquote(url)
-            if "$skiptoken=" in unquoted_url:
+            if "$skiptoken=" in unquoted_url and self.type != "PAGINATION_INFINITE_LOOP":
                 parsed_url = urlparse.urlparse(unquoted_url)
                 params = parse_qs(parsed_url.query)
                 file_name += "_" + params["$skiptoken"][0].split("!")[0]


[libcloud] 03/08: Bump the pagination limit to 300s.

Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git

commit cc30859500ec1781accbf85f18b0e61ee8bb7e0a
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Mon Jul 31 15:55:14 2023 +0200

    Bump the pagination limit to 300s.
---
 libcloud/compute/drivers/azure_arm.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libcloud/compute/drivers/azure_arm.py b/libcloud/compute/drivers/azure_arm.py
index 82273c565..678f6ae6d 100644
--- a/libcloud/compute/drivers/azure_arm.py
+++ b/libcloud/compute/drivers/azure_arm.py
@@ -66,7 +66,7 @@ VM_SIZE_API_VERSION = "2015-06-15"  # this API is deprecated
 
 # If pagination code in the list_nodes() method has still not completed after this mount of
 # seconds, we will break early from while True loop to avoid infinite loop under edge conditions.
-LIST_NODES_PAGINATION_TIMEOUT = 60
+LIST_NODES_PAGINATION_TIMEOUT = 300
 
 
 class AzureImage(NodeImage):