You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by ad...@apache.org on 2019/01/03 23:12:19 UTC
kudu git commit: python: fix tests run in Python 3.4 environments
Repository: kudu
Updated Branches:
refs/heads/master 8864cb284 -> 3ce9af404
python: fix tests run in Python 3.4 environments
Our Python build installs pandas, which is used in some tests. If we first
install numpy, pandas will use it for its build. This was the case in the
Python 2 variant of the Kudu build, which specified numpy 1.14, the last
version of numpy that was Python 2.6 compatible.
The Python 3 variant didn't explicitly install numpy, and thus was using
numpy 1.15.4 for quite some time. A few weeks ago numpy 1.16.0rc1 was
released, with one notable anti-feature: loss of Python 3.4 compatibility.
This is the version of Python 3 in Ubuntu 14.04; Python 3 tests run in
such environments would fail [1] like so:
../py_env/lib/python3.4/site-packages/py/_path/local.py:668: in pyimport
__import__(modname)
kudu/__init__.py:18: in <module>
from kudu.client import (Client, Table, Scanner, Session, # noqa
kudu/client.pyx:50: in init kudu.client
import pandas
../py_env/lib/python3.4/site-packages/pandas/__init__.py:26: in <module>
from pandas._libs import (hashtable as _hashtable,
../py_env/lib/python3.4/site-packages/pandas/_libs/__init__.py:4: in <module>
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/src/numpy.pxd:865: in init pandas._libs.tslib
???
E ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216
The fix is simple: since neither we nor pandas care much about the version
of numpy used, let's explicitly install the last Python 3.4 compatible numpy
in all Python 3 builds.
1. The breakage was not universal, likely due to wheel caching on pypi as
well as in the local pip cache.
Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Reviewed-on: http://gerrit.cloudera.org:8080/12148
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/3ce9af40
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/3ce9af40
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/3ce9af40
Branch: refs/heads/master
Commit: 3ce9af404a1081b02404316ab49b4a323bedbaf7
Parents: 8864cb2
Author: Adar Dembo <ad...@cloudera.com>
Authored: Wed Jan 2 17:19:46 2019 -0800
Committer: Adar Dembo <ad...@cloudera.com>
Committed: Thu Jan 3 23:12:04 2019 +0000
----------------------------------------------------------------------
build-support/jenkins/build-and-test.sh | 56 +++++++++++++++++-----------
1 file changed, 35 insertions(+), 21 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kudu/blob/3ce9af40/build-support/jenkins/build-and-test.sh
----------------------------------------------------------------------
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index b74c2ef..991d6ea 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -449,19 +449,16 @@ if [ "$BUILD_PYTHON" == "1" ]; then
# check disabled.
pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >= 0.8'
- # One of our dependencies is pandas, installed via requirements.txt below. It
- # depends on numpy, and if we don't install numpy directly, the pandas
- # installation will install the latest numpy which is incompatible with Python 2.6.
+ # One of our dependencies is pandas, installed below. It depends on numpy, and
+ # if we don't install numpy directly, the pandas installation will install the
+ # latest numpy which is incompatible with Python 2.6.
#
# To work around this, we need to install a 2.6-compatible version of numpy
- # before installing pandas. Listing such a numpy version in requirements.txt
- # doesn't work; it needs to be explicitly installed here.
- #
- # Installing numpy may involve some compiler work, so we must pass in the
- # current values of CC and CXX.
+ # before installing pandas. Installing numpy may involve some compiler work,
+ # so we must pass in the current values of CC and CXX.
#
# See https://github.com/numpy/numpy/releases/tag/v1.12.0 for more details.
- CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy <1.12.0'
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy < 1.12.0'
# We've got a new pip and new setuptools. We can now install the rest of the
# Python client's requirements.
@@ -470,13 +467,15 @@ if [ "$BUILD_PYTHON" == "1" ]; then
# pass in the current values of CC and CXX.
CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
- # We need to install Pandas manually since its not a required package but is needed
- # to run all of the tests.
- # pandas 0.18 dropped support for python 2.6.
+ # We need to install Pandas manually because although it's not a required
+ # package, it is needed to run all of the tests.
#
- # See https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016
+ # Installing pandas may involve some compiler work, so we must pass in the
+ # current values of CC and CXX.
+ #
+ # pandas 0.18 dropped support for python 2.6. See https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016
# for more details.
- pip install $PIP_INSTALL_FLAGS "pandas<0.18"
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'pandas < 0.18'
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
@@ -485,8 +484,8 @@ if [ "$BUILD_PYTHON" == "1" ]; then
CC=$CLANG CXX=$CLANG++ python setup.py build_ext
set +e
- # Run the Python tests.
- if ! python setup.py test \
+ # Run the Python tests. This may also involve some compiler work.
+ if ! CC=$CLANG CXX=$CLANG++ python setup.py test \
--addopts="kudu --junit-xml=$TEST_LOGDIR/python_client.xml" \
2> $TEST_LOGDIR/python_client.log ; then
TESTS_FAILED=1
@@ -529,6 +528,18 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
# check disabled.
pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >= 0.8'
+ # One of our dependencies is pandas, installed below. It depends on numpy, and
+ # if we don't install numpy directly, the pandas installation will install the
+ # latest numpy which is incompatible with Python 3.4 (the version of Python 3
+ # shipped with Ubuntu 14.04).
+ #
+ # To work around this, we need to install a 3.4-compatible version of numpy
+ # before installing pandas. Installing numpy may involve some compiler work,
+ # so we must pass in the current values of CC and CXX.
+ #
+ # See https://github.com/numpy/numpy/releases/tag/v1.16.0rc1 for more details.
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy < 1.16.0'
+
# We've got a new pip and new setuptools. We can now install the rest of the
# Python client's requirements.
#
@@ -536,9 +547,12 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
# pass in the current values of CC and CXX.
CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
- # We need to install Pandas manually since its not a required package but is needed
- # to run all of the tests.
- pip install $PIP_INSTALL_FLAGS pandas
+ # We need to install Pandas manually because although it's not a required
+ # package, it is needed to run all of the tests.
+ #
+ # Installing pandas may involve some compiler work, so we must pass in the
+ # current values of CC and CXX.
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS pandas
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
@@ -547,8 +561,8 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
CC=$CLANG CXX=$CLANG++ python setup.py build_ext
set +e
- # Run the Python tests.
- if ! python setup.py test \
+ # Run the Python tests. This may also involve some compiler work.
+ if ! CC=$CLANG CXX=$CLANG++ python setup.py test \
--addopts="kudu --junit-xml=$TEST_LOGDIR/python3_client.xml" \
2> $TEST_LOGDIR/python3_client.log ; then
TESTS_FAILED=1