You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@libcloud.apache.org by to...@apache.org on 2019/12/21 11:45:40 UTC
[libcloud] branch trunk updated (4107267 -> dc494b5)
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git.
from 4107267 Only pass "auth" argument to "create_node()" if that argument is passed to "deploy_node()" method.
new c8ae283 Add type annotations for more methods on the NodeDriver class.
new d23db72 For easier debugging and troubleshooting add __repr__ and __str__ to ScriptDeployment class.
new b8ea9fe Update example to also print out the output of the deployment script.
new 33da923 Fix formatting.
new b5ebd83 Add a note on paramiko logger opening log file in append mode which won't work in all the distros.
new 388ae32 Add changelog entry.
new 6d922eb Improve __repr__ if script didn't run yet.
new e0a762f Fix lint.
new dc494b5 Add some docs on how to access the output of a deployment step.
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.rst | 4 ++++
docs/compute/deployment.rst | 20 ++++++++++++++++-
docs/examples/compute/gce/deploy_node.py | 31 ++++++++++++++++----------
docs/troubleshooting.rst | 9 ++++++++
libcloud/compute/base.py | 38 +++++++++++++++++++++++++-------
libcloud/compute/deployment.py | 17 ++++++++++++++
6 files changed, 98 insertions(+), 21 deletions(-)
[libcloud] 05/09: Add a note on paramiko logger opening log file in
append mode which won't work in all the distros.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit b5ebd83476ba515c237118242d2823f9b3e9ba6b
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:27:08 2019 +0100
Add a note on paramiko logger opening log file in append mode which
won't work in all the distros.
---
docs/troubleshooting.rst | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/docs/troubleshooting.rst b/docs/troubleshooting.rst
index 8c12103..f4b299a 100644
--- a/docs/troubleshooting.rst
+++ b/docs/troubleshooting.rst
@@ -41,6 +41,15 @@ only works for non-chunked responses.
Example 1 - Logging output to standard error
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+.. note::
+
+ Some Linux distributions don't allow /dev/{stderr,stdout} to be opened in
+ append mode which means paramiko logger will return and error that it
+ can't log to that file.
+ In such scenario, it's recommended you specify actual file path and then
+ you can use "tail -F" in another terminal window to view the output in
+ real-time.
+
If you want the output to be logged to the standard error (on
Linux) you can set it to ``/dev/stderr``:
[libcloud] 03/09: Update example to also print out the output of
the deployment script.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit b8ea9fe0c2a4e274edacf41cf24e9cc19dc13c1e
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:22:48 2019 +0100
Update example to also print out the output of the deployment script.
---
docs/examples/compute/gce/deploy_node.py | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
diff --git a/docs/examples/compute/gce/deploy_node.py b/docs/examples/compute/gce/deploy_node.py
index 9ea4cab..dc21763 100644
--- a/docs/examples/compute/gce/deploy_node.py
+++ b/docs/examples/compute/gce/deploy_node.py
@@ -23,21 +23,22 @@ SERVICE_ACCOUNT_CREDENTIALS_JSON_FILE_PATH = '/path/to/sac.json'
PROJECT_ID = 'my-gcp-project'
Driver = get_driver(Provider.GCE)
-conn = Driver(SERVICE_ACCOUNT_USERNAME,
+driver = Driver(SERVICE_ACCOUNT_USERNAME,
SERVICE_ACCOUNT_CREDENTIALS_JSON_FILE_PATH,
project=PROJECT_ID,
datacenter='us-central1-a')
-# NOTE: To view the output of this command, you need to set
-# LIBCLOUD_DEBUG variable and then tail that file for the log output
step = ScriptDeployment("echo whoami ; date ; ls -la")
-images = conn.list_images()
-sizes = conn.list_sizes()
+images = driver.list_images()
+sizes = driver.list_sizes()
image = [i for i in images if i.name == 'ubuntu-1604-xenial-v20191217'][0]
size = [s for s in sizes if s.name == 'e2-micro'][0]
+print('Using image: %s' % (image))
+print('Using size: %s' % (size))
+
# NOTE: We specify which public key is installed on the instance using
# metadata functionality.
# Keep in mind that this step is only needed if you want to install a specific
@@ -56,7 +57,14 @@ ex_metadata = metadata = {
}
# deploy_node takes the same base keyword arguments as create_node.
-node = conn.deploy_node(name='libcloud-deploy-demo-1', image=image, size=size,
+node = driver.deploy_node(name='libcloud-deploy-demo-1', image=image, size=size,
ex_metadata=metadata,
deploy=step, ssh_key=PRIVATE_SSH_KEY_PATH)
-print(node)
+
+print('')
+print('Node: %s' % (node))
+print('')
+print('stdout: %s' % (step))
+print('stdout: %s' % (step.stdout))
+print('stderr: %s' % (step.stderr))
+print('exit_code: %s' % (step.exit_status))
[libcloud] 01/09: Add type annotations for more methods on the
NodeDriver class.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit c8ae2839cfa6ed42db7b333402d5b299e3a9e30f
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:00:50 2019 +0100
Add type annotations for more methods on the NodeDriver class.
---
libcloud/compute/base.py | 38 ++++++++++++++++++++++++++++++--------
1 file changed, 30 insertions(+), 8 deletions(-)
diff --git a/libcloud/compute/base.py b/libcloud/compute/base.py
index 06e1a2d..35a0b82 100644
--- a/libcloud/compute/base.py
+++ b/libcloud/compute/base.py
@@ -48,6 +48,7 @@ if TYPE_CHECKING:
from libcloud.compute.types import Provider
from libcloud.compute.types import NodeImageMemberState
from libcloud.compute.ssh import SSHClient
+from libcloud.compute.ssh import BaseSSHClient
from libcloud.common.base import Connection
from libcloud.common.base import ConnectionKey
from libcloud.common.base import BaseDriver
@@ -69,6 +70,8 @@ else:
T_Auth = Union['NodeAuthSSHKey', 'NodeAuthPassword']
+T_Ssh_key = Union[List[str], str]
+
# How long to wait for the node to come online after creating it
NODE_ONLINE_WAIT_TIMEOUT = 10 * 60
@@ -940,7 +943,7 @@ class NodeDriver(BaseDriver):
ssh_alternate_usernames=None, # type: Optional[List[str]]
ssh_port=22, # type: int
ssh_timeout=10, # type: int
- ssh_key=None, # type: Optional[str]
+ ssh_key=None, # type: Optional[T_Ssh_key]
auth=None, # type: T_Auth
timeout=SSH_CONNECT_TIMEOUT, # type: int
max_tries=3, # type: int
@@ -1509,9 +1512,14 @@ class NodeDriver(BaseDriver):
raise NotImplementedError(
'delete_key_pair not implemented for this driver')
- def wait_until_running(self, nodes, wait_period=3,
- timeout=600, ssh_interface='public_ips',
- force_ipv4=True, ex_list_nodes_kwargs=None):
+ def wait_until_running(self,
+ nodes, # type: List[Node]
+ wait_period=3, # type: float
+ timeout=600, # type: int
+ ssh_interface='public_ips', # type: str
+ force_ipv4=True, # type: bool
+ ex_list_nodes_kwargs=None # type: Optional[Dict]
+ ):
# type: (...) -> List[Tuple[Node, List[str]]]
"""
Block until the provided nodes are considered running.
@@ -1653,6 +1661,7 @@ class NodeDriver(BaseDriver):
def _wait_until_running(self, node, wait_period=3, timeout=600,
ssh_interface='public_ips', force_ipv4=True):
+ # type: (Node, float, int, str, bool) -> List[Tuple[Node, List[str]]]
# This is here for backward compatibility and will be removed in the
# next major release
return self.wait_until_running(nodes=[node], wait_period=wait_period,
@@ -1661,6 +1670,7 @@ class NodeDriver(BaseDriver):
force_ipv4=force_ipv4)
def _ssh_client_connect(self, ssh_client, wait_period=1.5, timeout=300):
+ # type: (BaseSSHClient, float, int) -> BaseSSHClient
"""
Try to connect to the remote SSH server. If a connection times out or
is refused it is retried up to timeout number of seconds.
@@ -1713,10 +1723,19 @@ class NodeDriver(BaseDriver):
raise LibcloudError(value='Could not connect to the remote SSH ' +
'server. Giving up.', driver=self)
- def _connect_and_run_deployment_script(self, task, node, ssh_hostname,
- ssh_port, ssh_username,
- ssh_password, ssh_key_file,
- ssh_timeout, timeout, max_tries):
+ def _connect_and_run_deployment_script(
+ self,
+ task, # type: Deployment
+ node, # type: Node
+ ssh_hostname, # type: str
+ ssh_port, # type: int
+ ssh_username, # type: str
+ ssh_password, # type: Optional[str]
+ ssh_key_file, # type:Optional[T_Ssh_key]
+ ssh_timeout, # type: int
+ timeout, # type: int
+ max_tries # type: int
+ ):
"""
Establish an SSH connection to the node and run the provided deployment
task.
@@ -1740,6 +1759,7 @@ class NodeDriver(BaseDriver):
return node
def _run_deployment_script(self, task, node, ssh_client, max_tries=3):
+ # type: (Deployment, Node, BaseSSHClient, int) -> Node
"""
Run the deployment script on the provided node. At this point it is
assumed that SSH connection has already been established.
@@ -1776,6 +1796,8 @@ class NodeDriver(BaseDriver):
ssh_client.close()
return node
+ return node
+
def _get_size_price(self, size_id):
"""
Return pricing information for the provided size id.
[libcloud] 09/09: Add some docs on how to access the output of a
deployment step.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit dc494b582d8e90b456d5d6c49c54ea90548232db
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:39:06 2019 +0100
Add some docs on how to access the output of a deployment step.
---
docs/compute/deployment.rst | 20 +++++++++++++++++++-
docs/examples/compute/gce/deploy_node.py | 1 -
2 files changed, 19 insertions(+), 2 deletions(-)
diff --git a/docs/compute/deployment.rst b/docs/compute/deployment.rst
index 0270d26..f9b1cc5 100644
--- a/docs/compute/deployment.rst
+++ b/docs/compute/deployment.rst
@@ -117,10 +117,28 @@ method and a couple of additional arguments. The most important ones are
authenticate. Key needs to be in a format which is supported by paramiko
(see section on supported key types above).
* ``ssh_username`` - SSH username used to login. If not provided, it defaults
- to ``root``
+ to ``root``.
* ``ssh_port`` - Port of the SSH server. If not provided, it defaults to
``22``.
+To view the output (stdout, stderr) and exit code of a specific deployment
+step, you can access ``stdout``, ``stderr`` and ``exit_status`` instance
+variables on the deployment class instance (``ScriptDeployment``,
+``ScriptFileDeployment``) in question.
+
+For example:
+
+.. sourcecode:: python
+
+ ...
+ step = ScriptDeployment("echo whoami ; date ; ls -la")
+
+ node = driver.deploy_node(...)
+
+ print('stdout: %s' % (step.stdout))
+ print('stderr: %s' % (step.stderr))
+ print('exit_code: %s' % (step.exit_status))
+
Some examples which demonstrate how this method can be used are displayed
below.
diff --git a/docs/examples/compute/gce/deploy_node.py b/docs/examples/compute/gce/deploy_node.py
index 10518ab..26cec6f 100644
--- a/docs/examples/compute/gce/deploy_node.py
+++ b/docs/examples/compute/gce/deploy_node.py
@@ -64,7 +64,6 @@ node = driver.deploy_node(name='libcloud-deploy-demo-1', image=image,
print('')
print('Node: %s' % (node))
print('')
-print('stdout: %s' % (step))
print('stdout: %s' % (step.stdout))
print('stderr: %s' % (step.stderr))
print('exit_code: %s' % (step.exit_status))
[libcloud] 08/09: Fix lint.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit e0a762faea5b9a5cdcaefb81d680cbef605d13a2
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:33:10 2019 +0100
Fix lint.
---
docs/examples/compute/gce/deploy_node.py | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/docs/examples/compute/gce/deploy_node.py b/docs/examples/compute/gce/deploy_node.py
index dc21763..10518ab 100644
--- a/docs/examples/compute/gce/deploy_node.py
+++ b/docs/examples/compute/gce/deploy_node.py
@@ -24,9 +24,9 @@ PROJECT_ID = 'my-gcp-project'
Driver = get_driver(Provider.GCE)
driver = Driver(SERVICE_ACCOUNT_USERNAME,
- SERVICE_ACCOUNT_CREDENTIALS_JSON_FILE_PATH,
- project=PROJECT_ID,
- datacenter='us-central1-a')
+ SERVICE_ACCOUNT_CREDENTIALS_JSON_FILE_PATH,
+ project=PROJECT_ID,
+ datacenter='us-central1-a')
step = ScriptDeployment("echo whoami ; date ; ls -la")
@@ -57,9 +57,9 @@ ex_metadata = metadata = {
}
# deploy_node takes the same base keyword arguments as create_node.
-node = driver.deploy_node(name='libcloud-deploy-demo-1', image=image, size=size,
- ex_metadata=metadata,
- deploy=step, ssh_key=PRIVATE_SSH_KEY_PATH)
+node = driver.deploy_node(name='libcloud-deploy-demo-1', image=image,
+ size=size, ex_metadata=metadata,
+ deploy=step, ssh_key=PRIVATE_SSH_KEY_PATH)
print('')
print('Node: %s' % (node))
[libcloud] 02/09: For easier debugging and troubleshooting add
__repr__ and __str__ to ScriptDeployment class.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit d23db725f5495a5d025efba9d8f2a3aaa7fd9e2d
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:17:45 2019 +0100
For easier debugging and troubleshooting add __repr__ and __str__ to
ScriptDeployment class.
---
libcloud/compute/deployment.py | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
diff --git a/libcloud/compute/deployment.py b/libcloud/compute/deployment.py
index 7dca950..306a6a2 100644
--- a/libcloud/compute/deployment.py
+++ b/libcloud/compute/deployment.py
@@ -209,6 +209,22 @@ class ScriptDeployment(Deployment):
return node
+ def __str__(self):
+ return self.__repr__()
+
+ def __repr__(self):
+ script = self.script[:15] + '...'
+ exit_status = self.exit_status
+
+ if exit_status is not None:
+ stdout = self.stdout[:30] + '...'
+ stderr = self.stderr[:30] + '...'
+ else:
+ stdout = None
+ stderr = None
+ return ("<ScriptDeployment script=%s, exit_status=%s, stdout=%s,"
+ "stderr=%s>" % (script, exit_status, stdout, stderr))
+
class ScriptFileDeployment(ScriptDeployment):
"""
[libcloud] 06/09: Add changelog entry.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 388ae328ff020079fbbb74cd701248c66372ef88
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:31:07 2019 +0100
Add changelog entry.
---
CHANGES.rst | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/CHANGES.rst b/CHANGES.rst
index 756e70b..8196fbc 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -69,6 +69,10 @@ Compute
(GITHUB-1387)
[Peter Yu - @yukw777, Tomaz Muraus]
+- [Common] To make debugging and troubleshooting easier, add ``__repr__``
+ and ``__str__`` method to the ``ScriptDeployment`` class.
+ [Tomaz Muraus]
+
Storage
-------
[libcloud] 04/09: Fix formatting.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 33da923b4b0ecfc3e08eac5b3fd566f398bede3f
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:23:51 2019 +0100
Fix formatting.
---
libcloud/compute/deployment.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/libcloud/compute/deployment.py b/libcloud/compute/deployment.py
index 306a6a2..cf9d5ff 100644
--- a/libcloud/compute/deployment.py
+++ b/libcloud/compute/deployment.py
@@ -222,7 +222,7 @@ class ScriptDeployment(Deployment):
else:
stdout = None
stderr = None
- return ("<ScriptDeployment script=%s, exit_status=%s, stdout=%s,"
+ return ("<ScriptDeployment script=%s, exit_status=%s, stdout=%s, "
"stderr=%s>" % (script, exit_status, stdout, stderr))
[libcloud] 07/09: Improve __repr__ if script didn't run yet.
Posted by to...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
tomaz pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/libcloud.git
commit 6d922eb88c9f86b2a3df33133c8530c3bcb63078
Author: Tomaz Muraus <to...@tomaz.me>
AuthorDate: Sat Dec 21 12:31:36 2019 +0100
Improve __repr__ if script didn't run yet.
---
libcloud/compute/deployment.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/libcloud/compute/deployment.py b/libcloud/compute/deployment.py
index cf9d5ff..0328725 100644
--- a/libcloud/compute/deployment.py
+++ b/libcloud/compute/deployment.py
@@ -220,6 +220,7 @@ class ScriptDeployment(Deployment):
stdout = self.stdout[:30] + '...'
stderr = self.stderr[:30] + '...'
else:
+ exit_status = 'script didn\'t run yet'
stdout = None
stderr = None
return ("<ScriptDeployment script=%s, exit_status=%s, stdout=%s, "