You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2018/07/16 23:26:55 UTC
[arrow] branch master updated: ARROW-2844: [Packaging] Test OSX
wheels after build
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new 56497ba ARROW-2844: [Packaging] Test OSX wheels after build
56497ba is described below
commit 56497badcfa4ef4b28c84eddbe2e41c6e20b6be6
Author: Krisztián Szűcs <sz...@gmail.com>
AuthorDate: Mon Jul 16 19:26:50 2018 -0400
ARROW-2844: [Packaging] Test OSX wheels after build
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #2272 from kszucs/ARROW-2844 and squashes the following commits:
3534722b <Krisztián Szűcs> don't forward positional arguments
d17ec106 <Krisztián Szűcs> update numpy build and test dependencies
f6b468bb <Krisztián Szűcs> remove parquet flag from pytest, success import of pyarrow.parquet triggers parquet tests
55a2a7aa <Krisztián Szűcs> fix multibuild_dir path
9092e39f <Krisztián Szűcs> remove surrounding whitespaces from bash variable assignment
b60040c6 <Krisztián Szűcs> couple of comments
732a5128 <Krisztián Szűcs> override multibuild's install_run
f48551f5 <Krisztián Szűcs> conda install location on appveyor is miniconda36 regardless the python version we build against
05f4c64a <Krisztián Szűcs> pass platform to install_run
ff75977e <Krisztián Szűcs> use multibuild's install_run instead
f4b1e016 <Krisztián Szűcs> fix requirements location
8e4847ba <Krisztián Szűcs> run tests against virtualenv installation
d40471bd <Krisztián Szűcs> too many popd
fc0b51c9 <Krisztián Szűcs> note about travis' auto cancellation
63d62874 <Krisztián Szűcs> update readme to install python 3.6
8c1d4e30 <Krisztián Szűcs> run pytest after build_wheel; sorted status output; added missing tasks from wheel group
---
dev/tasks/README.md | 12 ++++++++--
dev/tasks/conda-recipes/appveyor.yml | 4 +++-
dev/tasks/crossbow.py | 12 +++++-----
dev/tasks/python-wheels/osx-build.sh | 42 ++++++++++++++++++++++++++--------
dev/tasks/python-wheels/travis.osx.yml | 13 ++++++++---
dev/tasks/tasks.yml | 16 +++++++++----
6 files changed, 72 insertions(+), 27 deletions(-)
diff --git a/dev/tasks/README.md b/dev/tasks/README.md
index d7f0afa..9aaca1a 100644
--- a/dev/tasks/README.md
+++ b/dev/tasks/README.md
@@ -60,6 +60,9 @@ submission. The tasks are defined in `tasks.yml`
repository](https://help.github.com/articles/creating-a-new-repository)
2. Enable [TravisCI](https://travis-ci.org/getting_started) and
[Appveyor](https://www.appveyor.com/docs/) integrations on it
+
+ - turn off Travis' [auto cancellation](https://docs.travis-ci.com/user/customizing-the-build/#Building-only-the-latest-commit) feature on branches
+
3. Clone the newly created, by default the scripts looks for `crossbow` next to
arrow repository.
@@ -77,7 +80,12 @@ submission. The tasks are defined in `tasks.yml`
> or pass as an argument to the CLI script `--github-token`
-6. Install the python dependencies for the script:
+6. Install Python 3.6:
+
+ Miniconda is preferred, see installation instructions:
+ https://conda.io/docs/user-guide/install/index.html
+
+7. Install the python dependencies for the script:
```bash
conda install -y jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
@@ -88,7 +96,7 @@ submission. The tasks are defined in `tasks.yml`
pip install jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
```
-7. Try running it:
+8. Try running it:
```bash
$ python crossbow.py --help
```
diff --git a/dev/tasks/conda-recipes/appveyor.yml b/dev/tasks/conda-recipes/appveyor.yml
index 40436a6..47e331f 100644
--- a/dev/tasks/conda-recipes/appveyor.yml
+++ b/dev/tasks/conda-recipes/appveyor.yml
@@ -17,6 +17,8 @@
environment:
ARROW_VERSION: {{ arrow.version }}
+ # regardless of the python version we build against
+ CONDA_INSTALL_LOCN: C:\Miniconda36-x64
# We always use a 64-bit machine, but can build x86 distributions
# with the TARGET_ARCH variable.
@@ -28,7 +30,7 @@ install:
- cmd: rmdir C:\cygwin /s /q
# Add path, activate `conda` and update conda.
- - cmd: call {{ conda_install_location }}\Scripts\activate.bat
+ - cmd: call %CONDA_INSTALL_LOCN%\Scripts\activate.bat
- cmd: conda.exe update --yes --quiet conda
- cmd: set PYTHONUNBUFFERED=1
diff --git a/dev/tasks/crossbow.py b/dev/tasks/crossbow.py
index 0578e15..604f57f 100755
--- a/dev/tasks/crossbow.py
+++ b/dev/tasks/crossbow.py
@@ -65,7 +65,7 @@ class GitRemoteCallbacks(pygit2.RemoteCallbacks):
return None
-class Repo(object):
+class Repo:
"""Base class for interaction with local git repositories
A high level wrapper used for both reading revision information from
@@ -247,7 +247,7 @@ class Queue(Repo):
content_type=content_type)
-class Target(object):
+class Target:
"""Describes target repository and revision the builds run against
This serializable data container holding information about arrow's
@@ -273,7 +273,7 @@ class Target(object):
version=version)
-class Task(object):
+class Task:
"""Describes a build task and metadata required to render CI templates
A task is represented as a single git commit and branch containing jinja2
@@ -318,7 +318,7 @@ class Task(object):
return '.travis.yml'
-class Job(object):
+class Job:
"""Describes multiple tasks against a single target repository"""
def __init__(self, target, tasks):
@@ -469,7 +469,7 @@ def status(ctx, job_name):
job = queue.get(job_name)
statuses = queue.github_statuses(job)
- for task_name, task in job.tasks.items():
+ for task_name, task in sorted(job.tasks.items()):
status = statuses[task_name]
assets = queue.github_assets(task)
@@ -518,7 +518,7 @@ def sign(ctx, job_name, gpg_homedir, target_dir):
click.echo('Download {}\'s artifacts to {}'.format(job_name, target_dir))
tpl = '{:<10} {:>68}'
- for task_name, task in job.tasks.items():
+ for task_name, task in sorted(job.tasks.items()):
assets = queue.github_assets(task)
artifact_dir = target_dir / task_name
artifact_dir.mkdir(exist_ok=True)
diff --git a/dev/tasks/python-wheels/osx-build.sh b/dev/tasks/python-wheels/osx-build.sh
index 7bb1db1..6dce16f 100755
--- a/dev/tasks/python-wheels/osx-build.sh
+++ b/dev/tasks/python-wheels/osx-build.sh
@@ -19,6 +19,7 @@
set -e
+# overrides multibuild's default build_wheel
function build_wheel {
pip install -U pip
pip install setuptools_scm
@@ -191,18 +192,39 @@ function build_wheel {
--bundle-arrow-cpp --bundle-boost --boost-namespace=arrow_boost \
bdist_wheel
ls -l dist/
+ popd
+
+ popd
+}
+
+# overrides multibuild's default install_run
+function install_run {
+ multibuild_dir=`realpath $MULTIBUILD_DIR`
+
+ pushd $1 # enter arrow's directory
+
+ wheelhouse="$PWD/python/dist"
- # Do a test installation of the built wheel and change into another
- # directory to ensure our import tests later on pick up the wheel and
- # not the binaries from the build directory.
- for wheel in dist/*.whl; do
- pip install "$wheel"
- done
- mkdir -p tmp
- pushd tmp
+ # Install test dependencies and built wheel
+ if [ -n "$TEST_DEPENDS" ]; then
+ pip install $(pip_opts) $TEST_DEPENDS
+ fi
+ # Install compatible wheel
+ pip install $(pip_opts) \
+ $(python $multibuild_dir/supported_wheels.py $wheelhouse/*.whl)
+
+ # Runs tests on installed distribution from an empty directory
+ python --version
+
+ # Test optional dependencies
python -c "import pyarrow"
+ python -c "import pyarrow.orc"
python -c "import pyarrow.parquet"
- popd
- popd
+ python -c "import pyarrow.plasma"
+
+ # Run pyarrow tests
+ pip install pytest pytest-faulthandler
+ py.test --pyargs pyarrow
+
popd
}
diff --git a/dev/tasks/python-wheels/travis.osx.yml b/dev/tasks/python-wheels/travis.osx.yml
index d8a107b..fadc551 100644
--- a/dev/tasks/python-wheels/travis.osx.yml
+++ b/dev/tasks/python-wheels/travis.osx.yml
@@ -27,8 +27,8 @@ env:
- PYARROW_VERSION={{ arrow.version }}
- MACOSX_DEPLOYMENT_TARGET="10.9"
- MB_PYTHON_VERSION={{ python_version }}
- - NP_BUILD_DEP="{{ numpy_version }}"
- - NP_TEST_DEP="{{ numpy_version }}"
+ - NP_BUILD_DEP="{{ numpy_build_version }}"
+ - NP_TEST_DEP="{{ numpy_test_version }}"
- PANDAS_DEP="{{ pandas_version }}"
before_install:
@@ -53,8 +53,15 @@ install:
- TEST_DEPENDS="numpy==$NP_TEST_DEP six pandas==$PANDAS_DEP"
- mkdir -p dist
+
+ # multibuild's default clean_code function
- clean_code arrow $BUILD_REF
- - build_wheel arrow $PLAT
+
+ # the following functions are defined in osx-build.sh
+ - build_wheel arrow
+ - install_run arrow
+
+ # move built wheels to a top level directory
- mv -v arrow/python/dist/* dist/
deploy:
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 7171e70..53e149c 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -32,6 +32,11 @@ groups:
- wheel-linux-cp27mu
- wheel-linux-cp35m
- wheel-linux-cp36m
+ - wheel-osx-cp27m
+ - wheel-osx-cp35m
+ - wheel-osx-cp36m
+ - wheel-win-cp35m
+ - wheel-win-cp36m
linux:
- debian-stretch
- ubuntu-trusty
@@ -120,7 +125,6 @@ tasks:
platform: win
template: conda-recipes/appveyor.yml
params:
- conda_install_location: C:\\Miniconda35-x64
variant_config_file: variants\win_c_compilervs2015cxx_compilervs2015python3.5.yaml
artifacts:
- arrow-cpp-{version}-py35_vc14_0.tar.bz2
@@ -131,7 +135,6 @@ tasks:
platform: win
template: conda-recipes/appveyor.yml
params:
- conda_install_location: C:\\Miniconda36-x64
variant_config_file: variants\win_c_compilervs2015cxx_compilervs2015python3.6.yaml
artifacts:
- arrow-cpp-{version}-py36_vc14_0.tar.bz2
@@ -178,7 +181,8 @@ tasks:
platform: osx
template: python-wheels/travis.osx.yml
params:
- numpy_version: 1.10.1
+ numpy_build_version: 1.10.4
+ numpy_test_version: 1.14.0
pandas_version: 0.20.3
python_version: 2.7
artifacts:
@@ -188,7 +192,8 @@ tasks:
platform: osx
template: python-wheels/travis.osx.yml
params:
- numpy_version: 1.10.1
+ numpy_build_version: 1.10.4
+ numpy_test_version: 1.14.0
pandas_version: 0.20.3
python_version: 3.5
artifacts:
@@ -198,7 +203,8 @@ tasks:
platform: osx
template: python-wheels/travis.osx.yml
params:
- numpy_version: 1.11.3
+ numpy_build_version: 1.11.3
+ numpy_test_version: 1.14.0
pandas_version: 0.20.3
python_version: 3.6
artifacts: