You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Adar Dembo (Code Review)" <ge...@cloudera.org> on 2019/01/03 01:39:09 UTC
[kudu-CR] python: fix tests run in Python 3.4 environments
Hello andy@phdata.io, Alexey Serbin, Andrew Wong, Jordan Birdsell,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/12148
to review the following change.
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
python: fix tests run in Python 3.4 environments
Our Python build installs pandas, which is used in some tests. If we first
install numpy, pandas will use it for its build. This was the case in the
Python 2 variant of the Kudu build, which specified numpy 1.14, the last
version of numpy that was Python 2.6 compatible.
The Python 3 variant didn't explicitly install numpy, and thus was using
numpy 1.15.4 for quite some time. A few weeks ago numpy 1.16.0rc1 was
released, with one notable anti-feature: loss of Python 3.4 compatibility.
This is the version of Python 3 in Ubuntu 14.04; Python 3 tests run in
such environments would fail [1] like so:
../py_env/lib/python3.4/site-packages/py/_path/local.py:668: in pyimport
__import__(modname)
kudu/__init__.py:18: in <module>
from kudu.client import (Client, Table, Scanner, Session, # noqa
kudu/client.pyx:50: in init kudu.client
import pandas
../py_env/lib/python3.4/site-packages/pandas/__init__.py:26: in <module>
from pandas._libs import (hashtable as _hashtable,
../py_env/lib/python3.4/site-packages/pandas/_libs/__init__.py:4: in <module>
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/src/numpy.pxd:865: in init pandas._libs.tslib
???
E ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216
The fix is simple: since neither we nor pandas care much about the version
of numpy used, let's explicitly install the last Python 3.4 compatible numpy
in all Python 3 builds.
1. The breakage was not universal, likely due to wheel caching on pypi as
well as in the local pip cache.
Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
---
M build-support/jenkins/build-and-test.sh
1 file changed, 35 insertions(+), 21 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/48/12148/1
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
[kudu-CR] python: fix tests run in Python 3.4 environments
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12148 )
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12148/1/build-support/jenkins/build-and-test.sh
File build-support/jenkins/build-and-test.sh:
http://gerrit.cloudera.org:8080/#/c/12148/1/build-support/jenkins/build-and-test.sh@461
PS1, Line 461: CC=$CLANG CXX=$CLANG++
If everything that requires compilation needs to run with the current values of CC and CXX, would it make sense to export CC and CXX and unset/revert it once we're done?
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Jan 2019 20:15:10 +0000
Gerrit-HasComments: Yes
[kudu-CR] python: fix tests run in Python 3.4 environments
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12148 )
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
Patch Set 1: Code-Review+2
LGTM, but would be nice to get some input from the others.
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Jan 2019 22:08:57 +0000
Gerrit-HasComments: No
[kudu-CR] python: fix tests run in Python 3.4 environments
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12148 )
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
python: fix tests run in Python 3.4 environments
Our Python build installs pandas, which is used in some tests. If we first
install numpy, pandas will use it for its build. This was the case in the
Python 2 variant of the Kudu build, which specified numpy 1.14, the last
version of numpy that was Python 2.6 compatible.
The Python 3 variant didn't explicitly install numpy, and thus was using
numpy 1.15.4 for quite some time. A few weeks ago numpy 1.16.0rc1 was
released, with one notable anti-feature: loss of Python 3.4 compatibility.
This is the version of Python 3 in Ubuntu 14.04; Python 3 tests run in
such environments would fail [1] like so:
../py_env/lib/python3.4/site-packages/py/_path/local.py:668: in pyimport
__import__(modname)
kudu/__init__.py:18: in <module>
from kudu.client import (Client, Table, Scanner, Session, # noqa
kudu/client.pyx:50: in init kudu.client
import pandas
../py_env/lib/python3.4/site-packages/pandas/__init__.py:26: in <module>
from pandas._libs import (hashtable as _hashtable,
../py_env/lib/python3.4/site-packages/pandas/_libs/__init__.py:4: in <module>
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
pandas/_libs/src/numpy.pxd:865: in init pandas._libs.tslib
???
E ValueError: numpy.ufunc has the wrong size, try recompiling. Expected 192, got 216
The fix is simple: since neither we nor pandas care much about the version
of numpy used, let's explicitly install the last Python 3.4 compatible numpy
in all Python 3 builds.
1. The breakage was not universal, likely due to wheel caching on pypi as
well as in the local pip cache.
Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Reviewed-on: http://gerrit.cloudera.org:8080/12148
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <as...@cloudera.com>
---
M build-support/jenkins/build-and-test.sh
1 file changed, 35 insertions(+), 21 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 2
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] python: fix tests run in Python 3.4 environments
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12148 )
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Jan 2019 23:00:27 +0000
Gerrit-HasComments: No
[kudu-CR] python: fix tests run in Python 3.4 environments
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/12148 )
Change subject: python: fix tests run in Python 3.4 environments
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12148/1/build-support/jenkins/build-and-test.sh
File build-support/jenkins/build-and-test.sh:
http://gerrit.cloudera.org:8080/#/c/12148/1/build-support/jenkins/build-and-test.sh@461
PS1, Line 461: CC=$CLANG CXX=$CLANG++
> If everything that requires compilation needs to run with the current value
Maybe, but I like that this way we see which steps explicitly require compilation and which don't (though there's less and less of that now).
All of this is due for a rewrite; the lack of code sharing between the two Python steps is frustrating. I just didn't have the heart and decided to just apply yet another bandaid.
--
To view, visit http://gerrit.cloudera.org:8080/12148
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8683c6d24b3b808384a7e4d63142a78f3a7ee5f0
Gerrit-Change-Number: 12148
Gerrit-PatchSet: 1
Gerrit-Owner: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Anonymous Coward <an...@phdata.io>
Gerrit-Reviewer: Jordan Birdsell <jt...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 03 Jan 2019 21:27:30 +0000
Gerrit-HasComments: Yes