Repository: spark
Updated Branches:
refs/heads/master a763607e4 -> ff8de99a1
[SPARK-22302][INFRA] Remove manual backports for subprocess and print explicit message for < Python 2.7
## What changes were proposed in this pull request?
Seems there was a mistake - missing import for `subprocess.call`, while refactoring this script a long ago, which should be used for backports of some missing functions in `subprocess`, specifically in < Python 2.7.
Reproduction is:
```
cd dev && python2.6
```
```
>>> from sparktestsupport import shellutils
>>> shellutils.subprocess_check_call("ls")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sparktestsupport/shellutils.py", line 46, in subprocess_check_call
retcode = call(*popenargs, **kwargs)
NameError: global name 'call' is not defined
```
For Jenkins logs, please see https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3950/console
Since we dropped the Python 2.6.x support, looks better we remove those workarounds and print out explicit error messages in order to reduce the efforts to find out the root causes for such cases, for example, `https://github.com/apache/spark/pull/19513#issuecomment-337406734`.
## How was this patch tested?
Manually tested:
```
./dev/run-tests
```
```
Python versions prior to 2.7 are not supported.
```
```
./dev/run-tests-jenkins
```
```
Python versions prior to 2.7 are not supported.
```
Author: hyukjinkwon <gu...@gmail.com>
Closes #19524 from HyukjinKwon/SPARK-22302.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ff8de99a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ff8de99a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ff8de99a
Branch: refs/heads/master
Commit: ff8de99a1c7b4a291e661cd0ad12748f4321e43d
Parents: a763607
Author: hyukjinkwon <gu...@gmail.com>
Authored: Sun Oct 22 02:22:35 2017 +0900
Committer: hyukjinkwon <gu...@gmail.com>
Committed: Sun Oct 22 02:22:35 2017 +0900
----------------------------------------------------------------------
dev/run-tests | 6 ++++++
dev/run-tests-jenkins | 7 ++++++-
dev/sparktestsupport/shellutils.py | 31 ++-----------------------------
3 files changed, 14 insertions(+), 30 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/run-tests
----------------------------------------------------------------------
diff --git a/dev/run-tests b/dev/run-tests
index 257d1e8..9cf93d0 100755
--- a/dev/run-tests
+++ b/dev/run-tests
@@ -20,4 +20,10 @@
FWDIR="$(cd "`dirname $0`"/..; pwd)"
cd "$FWDIR"
+PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
+if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
+ echo "Python versions prior to 2.7 are not supported."
+ exit -1
+fi
+
exec python -u ./dev/run-tests.py "$@"
http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/run-tests-jenkins
----------------------------------------------------------------------
diff --git a/dev/run-tests-jenkins b/dev/run-tests-jenkins
index f41f1ac..03fd6ff 100755
--- a/dev/run-tests-jenkins
+++ b/dev/run-tests-jenkins
@@ -25,5 +25,10 @@
FWDIR="$( cd "$( dirname "$0" )/.." && pwd )"
cd "$FWDIR"
-export PATH=/home/anaconda/bin:$PATH
+PYTHON_VERSION_CHECK=$(python -c 'import sys; print(sys.version_info < (2, 7, 0))')
+if [[ "$PYTHON_VERSION_CHECK" == "True" ]]; then
+ echo "Python versions prior to 2.7 are not supported."
+ exit -1
+fi
+
exec python -u ./dev/run-tests-jenkins.py "$@"
http://git-wip-us.apache.org/repos/asf/spark/blob/ff8de99a/dev/sparktestsupport/shellutils.py
----------------------------------------------------------------------
diff --git a/dev/sparktestsupport/shellutils.py b/dev/sparktestsupport/shellutils.py
index 05af871..c7644da 100644
--- a/dev/sparktestsupport/shellutils.py
+++ b/dev/sparktestsupport/shellutils.py
@@ -21,35 +21,8 @@ import shutil
import subprocess
import sys
-
-if sys.version_info >= (2, 7):
- subprocess_check_output = subprocess.check_output
- subprocess_check_call = subprocess.check_call
-else:
- # SPARK-8763
- # backported from subprocess module in Python 2.7
- def subprocess_check_output(*popenargs, **kwargs):
- if 'stdout' in kwargs:
- raise ValueError('stdout argument not allowed, it will be overridden.')
- process = subprocess.Popen(stdout=subprocess.PIPE, *popenargs, **kwargs)
- output, unused_err = process.communicate()
- retcode = process.poll()
- if retcode:
- cmd = kwargs.get("args")
- if cmd is None:
- cmd = popenargs[0]
- raise subprocess.CalledProcessError(retcode, cmd, output=output)
- return output
-
- # backported from subprocess module in Python 2.7
- def subprocess_check_call(*popenargs, **kwargs):
- retcode = call(*popenargs, **kwargs)
- if retcode:
- cmd = kwargs.get("args")
- if cmd is None:
- cmd = popenargs[0]
- raise CalledProcessError(retcode, cmd)
- return 0
+subprocess_check_output = subprocess.check_output
+subprocess_check_call = subprocess.check_call
def exit_from_command_with_retcode(cmd, retcode):
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org