You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by pw...@apache.org on 2014/06/17 08:43:51 UTC
git commit: SPARK-1990: added compatibility for python 2.6 for
ssh_read command
Repository: spark
Updated Branches:
refs/heads/master d81c08bac -> 8cd04c3ee
SPARK-1990: added compatibility for python 2.6 for ssh_read command
https://issues.apache.org/jira/browse/SPARK-1990
There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old
Author: Anant <an...@gmail.com>
Closes #941 from anantasty/SPARK-1990 and squashes the following commits:
4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6
c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8cd04c3e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8cd04c3e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8cd04c3e
Branch: refs/heads/master
Commit: 8cd04c3eecc2dd827ea163dcd5e08af9912fa323
Parents: d81c08b
Author: Anant <an...@gmail.com>
Authored: Mon Jun 16 23:42:27 2014 -0700
Committer: Patrick Wendell <pw...@gmail.com>
Committed: Mon Jun 16 23:43:01 2014 -0700
----------------------------------------------------------------------
ec2/spark_ec2.py | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8cd04c3e/ec2/spark_ec2.py
----------------------------------------------------------------------
diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py
index 52a89cb..803caa0 100755
--- a/ec2/spark_ec2.py
+++ b/ec2/spark_ec2.py
@@ -689,9 +689,23 @@ def ssh(host, opts, command):
time.sleep(30)
tries = tries + 1
+# Backported from Python 2.7 for compatiblity with 2.6 (See SPARK-1990)
+def _check_output(*popenargs, **kwargs):
+ if 'stdout' in kwargs:
+ raise ValueError('stdout argument not allowed, it will be overridden.')
+ process = subprocess.Popen(stdout=PIPE, *popenargs, **kwargs)
+ output, unused_err = process.communicate()
+ retcode = process.poll()
+ if retcode:
+ cmd = kwargs.get("args")
+ if cmd is None:
+ cmd = popenargs[0]
+ raise subprocess.CalledProcessError(retcode, cmd, output=output)
+ return output
+
def ssh_read(host, opts, command):
- return subprocess.check_output(
+ return _check_output(
ssh_command(opts) + ['%s@%s' % (opts.user, host), stringify_command(command)])