You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by mp...@apache.org on 2019/02/28 01:13:09 UTC
[kudu] branch branch-1.8.x updated (94c1b70 -> 3a8ef69)
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a change to branch branch-1.8.x
in repository https://gitbox.apache.org/repos/asf/kudu.git.
from 94c1b70 KUDU-2710: Fix KeepAliveRequest retries
new e283181 python: fix Python 3.4 based tests
new 3a8ef69 python: fix tests run in Python 3.4 environments
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
build-support/jenkins/build-and-test.sh | 65 ++++++++++++++++++++++-----------
1 file changed, 43 insertions(+), 22 deletions(-)
[kudu] 01/02: python: fix Python 3.4 based tests
Posted by mp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a commit to branch branch-1.8.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit e28318120ce1fb7195c58a2f096903b32aafa257
Author: Hao Hao <ha...@cloudera.com>
AuthorDate: Tue Jan 22 18:08:07 2019 -0800
python: fix Python 3.4 based tests
Pip released 19.0 recently and tests based on Python 3.4 started to fail. In
addition, Python 3.4 will no longer be supported beginning with pip 19.1.
Therefore, this patch updates the pip version to the last one that works.
The full error message is below.
$ pip install pandas
DEPRECATION: Python 3.4 support has been deprecated. pip 19.1 will be the last one supporting it. Please upgrade your Python as Python 3.4 won't be maintained after March 2019 (cf PEP 429).
Collecting pandas
Using cached https://files.pythonhosted.org/packages/08/01/803834bc8a4e708aedebb133095a88a4dad9f45bbaf5ad777d2bea543c7e/pandas-0.22.0.tar.gz
Installing build dependencies ... done
Getting requirements to build wheel ... error
Complete output from command /home/jenkins-slave/a/bin/python3 /home/jenkins-slave/a/lib/python3.4/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpsknqc6o3:
Traceback (most recent call last):
File "/home/jenkins-slave/a/lib/python3.4/site-packages/pip/_vendor/pep517/_in_process.py", line 207, in <module>
main()
File "/home/jenkins-slave/a/lib/python3.4/site-packages/pip/_vendor/pep517/_in_process.py", line 197, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/jenkins-slave/a/lib/python3.4/site-packages/pip/_vendor/pep517/_in_process.py", line 54, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-z5ha9gs9/overlay/lib/python3.4/site-packages/setuptools/build_meta.py", line 115, in get_requires_for_build_wheel
return _get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-z5ha9gs9/overlay/lib/python3.4/site-packages/setuptools/build_meta.py", line 101, in _get_build_requires
_run_setup()
File "/tmp/pip-build-env-z5ha9gs9/overlay/lib/python3.4/site-packages/setuptools/build_meta.py", line 85, in _run_setup
exec(compile(code, __file__, 'exec'), locals())
File "setup.py", line 27, in <module>
import versioneer
ImportError: No module named 'versioneer'
Change-Id: Iecbc55877e832a0ef437d2799e09c39fb314cb95
Reviewed-on: http://gerrit.cloudera.org:8080/12255
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Adar Dembo <ad...@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/12626
Tested-by: Kudu Jenkins
---
build-support/jenkins/build-and-test.sh | 9 ++++++++-
1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index 6ba91c9..a308890 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -554,7 +554,14 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
# recursively to transitive dependencies installed via a direct dependency's
# "python setup.py" command. Therefore we have no choice but to upgrade to a
# new version of pip to proceed.
- pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade pip
+ #
+ # pip 19.1 doesn't support Python 3.4, which is the version of Python 3
+ # shipped with Ubuntu 14.04. However, there appears to be a bug[1] in pip 19.0
+ # preventing it from working properly with Python 3.4 as well. Therefore we
+ # must pin to a pip version from before 19.0.
+ #
+ # 1. https://github.com/pypa/pip/issues/6175
+ pip install -i https://pypi.python.org/simple $PIP_INSTALL_FLAGS --upgrade 'pip < 19.0'
# New versions of pip raise an exception when upgrading old versions of
# setuptools (such as the one found in el6). The workaround is to upgrade
[kudu] 02/02: python: fix tests run in Python 3.4 environments
Posted by mp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mpercy pushed a commit to branch branch-1.8.x
in repository https://gitbox.apache.org/repos/asf/kudu.git
commit 3a8ef69cdb20d23a98f645e86d9b383c20f24355
Author: Adar Dembo <ad...@cloudera.com>
AuthorDate: Wed Jan 2 17:19:46 2019 -0800
python: fix tests run in Python 3.4 environments
Our Python build installs pandas, which is used in some tests. If we first
install numpy, pandas will use it for its build. This was the case in the
Python 2 variant of the Kudu build, which specified numpy 1.14, the last
version of numpy that was Python 2.6 compatible.
The Python 3 variant didn't explicitly install numpy, and thus was using
numpy 1.15.4 for quite some time. A few weeks ago numpy 1.16.0rc1 was
released, with one notable anti-feature: loss of Python 3.4 compatibility.
This is the version of Python 3 in Ubuntu 14.04; Python 3 tests run in
such environments would fail [1] like so:
../py_env/lib/python3.4/site-packages/py/_path/local.py:668: in pyimport
__import__(modname)
kudu/__init__.py:18: in <module>
from kudu.client import (Client, Table, Scanner, Session, # noqa
kudu/client.pyx:50: in init kudu.client
import pandas
../py_env/lib/python3.4/site-packages/pandas/__init__.py:26: in <module>
from pandas._libs import (hashtable as _hashtable,
../py_env/lib/python3.4/site-packages/pandas/_libs/__init__.py:4: in <module>
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/src/numpy.pxd:865: in init pandas._libs.tslib
???
E ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216
The fix is simple: since neither we nor pandas care much about the version
of numpy used, let's explicitly install the last Python 3.4 compatible numpy
in all Python 3 builds.
1. The breakage was not universal, likely due to wheel caching on pypi as
well as in the local pip cache.
Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Reviewed-on: http://gerrit.cloudera.org:8080/12148
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Reviewed-on: http://gerrit.cloudera.org:8080/12625
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
build-support/jenkins/build-and-test.sh | 56 ++++++++++++++++++++-------------
1 file changed, 35 insertions(+), 21 deletions(-)
diff --git a/build-support/jenkins/build-and-test.sh b/build-support/jenkins/build-and-test.sh
index a308890..8de9023 100755
--- a/build-support/jenkins/build-and-test.sh
+++ b/build-support/jenkins/build-and-test.sh
@@ -482,19 +482,16 @@ if [ "$BUILD_PYTHON" == "1" ]; then
# check disabled.
pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >= 0.8'
- # One of our dependencies is pandas, installed via requirements.txt below. It
- # depends on numpy, and if we don't install numpy directly, the pandas
- # installation will install the latest numpy which is incompatible with Python 2.6.
+ # One of our dependencies is pandas, installed below. It depends on numpy, and
+ # if we don't install numpy directly, the pandas installation will install the
+ # latest numpy which is incompatible with Python 2.6.
#
# To work around this, we need to install a 2.6-compatible version of numpy
- # before installing pandas. Listing such a numpy version in requirements.txt
- # doesn't work; it needs to be explicitly installed here.
- #
- # Installing numpy may involve some compiler work, so we must pass in the
- # current values of CC and CXX.
+ # before installing pandas. Installing numpy may involve some compiler work,
+ # so we must pass in the current values of CC and CXX.
#
# See https://github.com/numpy/numpy/releases/tag/v1.12.0 for more details.
- CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy <1.12.0'
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy < 1.12.0'
# We've got a new pip and new setuptools. We can now install the rest of the
# Python client's requirements.
@@ -503,13 +500,15 @@ if [ "$BUILD_PYTHON" == "1" ]; then
# pass in the current values of CC and CXX.
CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
- # We need to install Pandas manually since its not a required package but is needed
- # to run all of the tests.
- # pandas 0.18 dropped support for python 2.6.
+ # We need to install Pandas manually because although it's not a required
+ # package, it is needed to run all of the tests.
#
- # See https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016
+ # Installing pandas may involve some compiler work, so we must pass in the
+ # current values of CC and CXX.
+ #
+ # pandas 0.18 dropped support for python 2.6. See https://pandas.pydata.org/pandas-docs/version/0.23.0/whatsnew.html#v0-18-0-march-13-2016
# for more details.
- pip install $PIP_INSTALL_FLAGS "pandas<0.18"
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'pandas < 0.18'
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
@@ -518,8 +517,8 @@ if [ "$BUILD_PYTHON" == "1" ]; then
CC=$CLANG CXX=$CLANG++ python setup.py build_ext
set +e
- # Run the Python tests.
- if ! python setup.py test \
+ # Run the Python tests. This may also involve some compiler work.
+ if ! CC=$CLANG CXX=$CLANG++ python setup.py test \
--addopts="kudu --junit-xml=$TEST_LOGDIR/python_client.xml" \
2> $TEST_LOGDIR/python_client.log ; then
TESTS_FAILED=1
@@ -569,6 +568,18 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
# check disabled.
pip install --disable-pip-version-check $PIP_INSTALL_FLAGS --upgrade 'setuptools >= 0.8'
+ # One of our dependencies is pandas, installed below. It depends on numpy, and
+ # if we don't install numpy directly, the pandas installation will install the
+ # latest numpy which is incompatible with Python 3.4 (the version of Python 3
+ # shipped with Ubuntu 14.04).
+ #
+ # To work around this, we need to install a 3.4-compatible version of numpy
+ # before installing pandas. Installing numpy may involve some compiler work,
+ # so we must pass in the current values of CC and CXX.
+ #
+ # See https://github.com/numpy/numpy/releases/tag/v1.16.0rc1 for more details.
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS 'numpy < 1.16.0'
+
# We've got a new pip and new setuptools. We can now install the rest of the
# Python client's requirements.
#
@@ -576,9 +587,12 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
# pass in the current values of CC and CXX.
CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS -r requirements.txt
- # We need to install Pandas manually since its not a required package but is needed
- # to run all of the tests.
- pip install $PIP_INSTALL_FLAGS pandas
+ # We need to install Pandas manually because although it's not a required
+ # package, it is needed to run all of the tests.
+ #
+ # Installing pandas may involve some compiler work, so we must pass in the
+ # current values of CC and CXX.
+ CC=$CLANG CXX=$CLANG++ pip install $PIP_INSTALL_FLAGS pandas
# Delete old Cython extensions to force them to be rebuilt.
rm -Rf build kudu_python.egg-info kudu/*.so
@@ -587,8 +601,8 @@ if [ "$BUILD_PYTHON3" == "1" ]; then
CC=$CLANG CXX=$CLANG++ python setup.py build_ext
set +e
- # Run the Python tests.
- if ! python setup.py test \
+ # Run the Python tests. This may also involve some compiler work.
+ if ! CC=$CLANG CXX=$CLANG++ python setup.py test \
--addopts="kudu --junit-xml=$TEST_LOGDIR/python3_client.xml" \
2> $TEST_LOGDIR/python3_client.log ; then
TESTS_FAILED=1