You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2018/07/10 18:07:58 UTC
[arrow] branch master updated: ARROW-2724: [Packaging] Determine
whether all the expected artifacts are uploaded
This is an automated email from the ASF dual-hosted git repository.
wesm pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new bd1c332 ARROW-2724: [Packaging] Determine whether all the expected artifacts are uploaded
bd1c332 is described below
commit bd1c33265c0580e59f35317598f3035fb9065908
Author: Krisztián Szűcs <sz...@gmail.com>
AuthorDate: Tue Jul 10 14:07:52 2018 -0400
ARROW-2724: [Packaging] Determine whether all the expected artifacts are uploaded
Also resolves [ARROW-2718: [Packaging] GPG sign downloaded artifacts](https://issues.apache.org/jira/browse/ARROW-2718)
Contains a sample configuration for running nightly builds.
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #2162 from kszucs/ARROW-2724 and squashes the following commits:
2f0cb9b1 <Krisztián Szűcs> increase travis timeout for debian builds
b4303f77 <Krisztián Szűcs> fix artifact upload on conda-win
eb06a0d0 <Krisztián Szűcs> don't install conda-forge-ci-setup on linux
6c384254 <Krisztián Szűcs> converge to conda-forge-ci-setup
3c3967a5 <Krisztián Szűcs> conda build fixes
8b9a05ae <Krisztián Szűcs> fix conda build variant paths on appveyoer builds
c5d158cd <Krisztián Szűcs> added conda build variants; incorparate recipe changes from upstream feedstocks
27d635ec <Krisztián Szűcs> ubuntu-bionic
0afc59b1 <Krisztián Szűcs> docstrings
a858d2ad <Krisztián Szűcs> assign github token to repo instance
9e7f64a8 <Krisztián Szűcs> properly parse job idss from remote branch names
49d14f02 <Krisztián Szűcs> fetch all of the remote branches
461afe5e <Krisztián Szűcs> todo notes for resolving deb artifact naming conflicts
aa216fb1 <Krisztián Szűcs> remove date from linux package artifact names
d54ec49c <Krisztián Szűcs> little nicer dry-run output
3f9f9cd1 <Krisztián Szűcs> note about running nightlies against a fork
d693838b <Krisztián Szűcs> sample configuration for scheduling cron jobs on travis
54d3358f <Krisztián Szűcs> relative template paths to dev/tasks directory
63a0a7fb <Krisztián Szűcs> handle querying not yet created github release
e1b49400 <Krisztián Szűcs> define build artifacts for linux=pkgs
c4c90e82 <Krisztián Szűcs> try to read linux-pkgs version from env
516eead5 <Krisztián Szűcs> deploy .deb and .rpm packages
61d74108 <Krisztián Szűcs> linux packages timeout
f44d95ca <Krisztián Szűcs> yum orc
9b618458 <Krisztián Szűcs> dont upload signatures to github
e812399e <Krisztián Szűcs> turn off versbose 3rdparty log
1c851165 <Krisztián Szűcs> debian pkgconfig orc
a216e984 <Krisztián Szűcs> download and gpg sign artifacts; upload signatures to github releases
b86258c3 <Krisztián Szűcs> deb: reproducible timeless
c41748e7 <Krisztián Szűcs> group artifacts per task during download; fail on missing assets
c7394166 <Krisztián Szűcs> versose 3rdparty build
6ec941fb <Krisztián Szűcs> little status improvements
b6bb1618 <Krisztián Szűcs> update documentation to use ruamel yaml instead of pyyaml
12a8a524 <Krisztián Szűcs> remove unused queue prop
1878d007 <Krisztián Szűcs> rename artifacts command
be9dadf3 <Krisztián Szűcs> reduce boilerplate; define task artifacts explicitly; query artifacts' statuses
---
.gitignore | 1 +
dev/tasks/README.md | 4 +-
dev/tasks/conda-recipes/appveyor.yml | 31 +-
dev/tasks/conda-recipes/arrow-cpp/build.sh | 2 +-
dev/tasks/conda-recipes/arrow-cpp/meta.yaml | 21 +-
dev/tasks/conda-recipes/parquet-cpp/meta.yaml | 13 +-
dev/tasks/conda-recipes/pyarrow/meta.yaml | 16 +-
dev/tasks/conda-recipes/travis.linux.yml | 24 +-
dev/tasks/conda-recipes/travis.osx.yml | 25 +-
.../conda-recipes/variants/linux_python2.7.yaml | 52 ++-
.../conda-recipes/variants/linux_python3.5.yaml | 52 ++-
.../conda-recipes/variants/linux_python3.6.yaml | 52 ++-
.../conda-recipes/variants/osx_python2.7.yaml | 58 ++-
.../conda-recipes/variants/osx_python3.5.yaml | 58 ++-
.../conda-recipes/variants/osx_python3.6.yaml | 52 ++-
..._compilervs2015cxx_compilervs2015python3.5.yaml | 56 ++-
..._compilervs2015cxx_compilervs2015python3.6.yaml | 56 ++-
dev/tasks/config-nightlies/travis.linux.yml | 67 ---
dev/tasks/crossbow.py | 460 +++++++++++++--------
dev/tasks/linux-packages/Rakefile | 1 +
.../apt/ubuntu-bionic/Dockerfile} | 52 +--
.../debian.ubuntu-trusty/libarrow-dev.install | 1 +
.../linux-packages/debian.ubuntu-trusty/rules | 2 +
.../linux-packages/debian/libarrow-dev.install | 1 +
dev/tasks/linux-packages/debian/rules | 4 +-
dev/tasks/linux-packages/package-task.rb | 1 +
dev/tasks/linux-packages/travis.linux.yml | 65 ++-
dev/tasks/linux-packages/yum/arrow.spec.in | 1 +
.../travis.linux.yml => nightlies.sample.yml} | 81 ++--
dev/tasks/python-wheels/appveyor.yml | 12 +-
dev/tasks/python-wheels/travis.linux.yml | 14 +-
dev/tasks/python-wheels/travis.osx.yml | 13 +-
dev/tasks/tasks.yml | 96 ++++-
33 files changed, 870 insertions(+), 574 deletions(-)
diff --git a/.gitignore b/.gitignore
index d9c69e9..ab51a6f 100644
--- a/.gitignore
+++ b/.gitignore
@@ -37,3 +37,4 @@ python/.eggs/
.vscode
.idea/
.pytest_cache/
+pkgs
diff --git a/dev/tasks/README.md b/dev/tasks/README.md
index 35140fa..ecf690b 100644
--- a/dev/tasks/README.md
+++ b/dev/tasks/README.md
@@ -80,12 +80,12 @@ submission. The tasks are defined in `tasks.yml`
6. Install the python dependencies for the script:
```bash
- conda install -y jinja2 pygit2 click pyyaml setuptools_scm github3.py
+ conda install -y jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
```
```bash
# pygit2 requires libgit2: http://www.pygit2.org/install.html
- pip install -y jinja2 pygit2 click pyyaml setuptools_scm github3.py
+ pip install -y jinja2 pygit2 click ruamel.yaml setuptools_scm github3.py python-gnupg
```
7. Try running it:
diff --git a/dev/tasks/conda-recipes/appveyor.yml b/dev/tasks/conda-recipes/appveyor.yml
index 3d3f330..664c0b2 100644
--- a/dev/tasks/conda-recipes/appveyor.yml
+++ b/dev/tasks/conda-recipes/appveyor.yml
@@ -16,38 +16,47 @@
# under the License.
environment:
+ ARROW_VERSION: {{ arrow.version }}
matrix:
- - TARGET_ARCH: x64
- CONDA_PY: 35
+ - CONFIG: variants\win_c_compilervs2015cxx_compilervs2015python3.5.yaml
CONDA_INSTALL_LOCN: C:\\Miniconda35-x64
- - TARGET_ARCH: x64
- CONDA_PY: 36
+
+ - CONFIG: variants\win_c_compilervs2015cxx_compilervs2015python3.6.yaml
CONDA_INSTALL_LOCN: C:\\Miniconda36-x64
- ARROW_VERSION: {{ ARROW_VERSION }}
+
+# We always use a 64-bit machine, but can build x86 distributions
+# with the TARGET_ARCH variable.
+platform:
+ - x64
install:
+ # Cywing's git breaks conda-build. (See https://github.com/conda-forge/conda-smithy-feedstock/pull/2.)
- cmd: rmdir C:\cygwin /s /q
+
# Add path, activate `conda` and update conda.
- cmd: call %CONDA_INSTALL_LOCN%\Scripts\activate.bat
- cmd: conda.exe update --yes --quiet conda
+
- cmd: set PYTHONUNBUFFERED=1
+
# Add our channels.
- cmd: conda.exe config --set show_channel_urls true
- cmd: conda.exe config --remove channels defaults
- cmd: conda.exe config --add channels defaults
- cmd: conda.exe config --add channels conda-forge
+
# Configure the VM.
- - cmd: conda.exe install -n root --quiet --yes conda-forge-build-setup
+ - cmd: conda.exe install -n root --quiet --yes conda-forge-ci-setup=1
- cmd: run_conda_forge_build_setup
# Skip .NET project specific build phase.
build: off
test_script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow || exit /B
- - git -C arrow checkout {{ ARROW_SHA }} || exit /B
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow || exit /B
+ - git -C arrow checkout {{ arrow.head }} || exit /B
- pushd arrow\dev\tasks\conda-recipes
- - conda.exe build --output-folder . parquet-cpp arrow-cpp pyarrow
+ - conda.exe build --output-folder . -m %CONFIG% parquet-cpp arrow-cpp pyarrow
- pushd win-64
- for %%f in (*.tar.bz2) do (
set %%g=%%~nf
@@ -59,7 +68,7 @@ artifacts:
- path: arrow\dev\tasks\conda-recipes\win-64\*.tar.bz2
deploy:
- release: {{ BUILD_TAG }}
+ release: {{ job.branch }}
provider: GitHub
auth_token: "%CROSSBOW_GITHUB_TOKEN%"
artifact: /.*\.tar\.bz2/
@@ -70,4 +79,4 @@ deploy:
notifications:
- provider: Email
to:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/conda-recipes/arrow-cpp/build.sh b/dev/tasks/conda-recipes/arrow-cpp/build.sh
index 7ec43cc..8b39764 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/build.sh
+++ b/dev/tasks/conda-recipes/arrow-cpp/build.sh
@@ -35,7 +35,7 @@ cmake \
-DARROW_BUILD_BENCHMARKS=OFF \
-DARROW_BUILD_UTILITIES=OFF \
-DARROW_BUILD_TESTS=OFF \
- -DARROW_JEMALLOC=OFF \
+ -DARROW_JEMALLOC=ON \
-DARROW_PLASMA=ON \
-DARROW_PYTHON=ON \
-DARROW_ORC=ON \
diff --git a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
index ae9f87d..6439ee3 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
+++ b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
@@ -31,27 +31,28 @@ build:
requirements:
build:
- - toolchain
- cmake
- - boost-cpp 1.66.0
+ - autoconf # [unix]
+ - {{ compiler('c') }}
+ - {{ compiler('cxx') }}
+ host:
+ - boost-cpp
- flatbuffers
- rapidjson
- - zlib 1.2.*
+ - zlib
- snappy
- brotli
- zstd
- lz4-c
- - jemalloc 4.4.0 # [unix]
- python
- - numpy 1.10.* # [not (win and py>=36)]
- - numpy 1.12.* # [win and py>=36]
+ - numpy 1.10.* # [not (win and py>=36)]
+ - numpy 1.12.* # [win and py>=36]
run:
- - boost-cpp 1.66.0
+ - boost-cpp
- python
- - numpy >=1.10 # [not (win and py>=36)]
- - numpy >=1.12 # [win and py>=36]
- - vc 14.* # [win and py>=35]
+ - {{ pin_compatible('numpy', lower_bound='1.10') }} # [not (win and py>=36)]
+ - {{ pin_compatible('numpy', lower_bound='1.12') }} # [win and py>=36]
- snappy # [win and py>=35]
test:
diff --git a/dev/tasks/conda-recipes/parquet-cpp/meta.yaml b/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
index e7be2a1..2b76a01 100644
--- a/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
+++ b/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
@@ -35,19 +35,22 @@ source:
sha256: {{ sha256sum }}
build:
- number: 0
- features:
- - vc14 # [win]
+ number: 2
+ skip: true # [win32]
+ skip: true # [win and py<35]
requirements:
build:
- - toolchain
- - boost-cpp 1.66.0
- cmake
+ - {{ compiler('c') }}
+ - {{ compiler('cxx') }}
+ host:
+ - boost-cpp
- thrift-cpp >=0.11
- arrow-cpp {{ ARROW_VERSION }}
run:
+ - boost-cpp
- arrow-cpp {{ ARROW_VERSION }}
test:
diff --git a/dev/tasks/conda-recipes/pyarrow/meta.yaml b/dev/tasks/conda-recipes/pyarrow/meta.yaml
index 2d797e5..c766903 100644
--- a/dev/tasks/conda-recipes/pyarrow/meta.yaml
+++ b/dev/tasks/conda-recipes/pyarrow/meta.yaml
@@ -26,19 +26,19 @@ build:
number: 0
skip: true # [win32]
skip: true # [win and py<35]
- features:
- - vc14 # [win and py>=35]
requirements:
build:
+ - cmake
+ - {{ compiler('c') }}
+ - {{ compiler('cxx') }}
+ host:
- python
- - toolchain
- setuptools
- setuptools_scm
- cython
- - cmake
- - numpy 1.10.* # [not (win and py>=36)]
- - numpy 1.12.* # [win and py>=36]
+ - numpy 1.10.* # [not (win and py>=36)]
+ - numpy 1.12.* # [win and py>=36]
- six
- arrow-cpp {{ ARROW_VERSION }}
- parquet-cpp 1.4.0
@@ -46,8 +46,8 @@ requirements:
run:
- python
- setuptools
- - numpy >=1.10 # [not (win and py>=36)]
- - numpy >=1.12 # [win and py>=36]
+ - {{ pin_compatible('numpy', lower_bound='1.10') }} # [not (win and py>=36)]
+ - {{ pin_compatible('numpy', lower_bound='1.12') }} # [win and py>=36]
- pandas
- six
- arrow-cpp {{ ARROW_VERSION }}
diff --git a/dev/tasks/conda-recipes/travis.linux.yml b/dev/tasks/conda-recipes/travis.linux.yml
index 84ca83b..7686c4f 100644
--- a/dev/tasks/conda-recipes/travis.linux.yml
+++ b/dev/tasks/conda-recipes/travis.linux.yml
@@ -22,12 +22,13 @@ dist: trusty
env:
matrix:
- - CONDA_PY=27
- - CONDA_PY=35
- - CONDA_PY=36
+ - CONFIG=variants/linux_python2.7.yaml
+ - CONFIG=variants/linux_python3.5.yaml
+ - CONFIG=variants/linux_python3.6.yaml
global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - ARROW_VERSION={{ ARROW_VERSION }}
+ - TRAVIS_TAG={{ job.branch }}
+ - ARROW_VERSION={{ arrow.version }}
+ - PYTHONUNBUFFERED=1
install:
# Install Miniconda.
@@ -49,16 +50,15 @@ install:
conda config --add channels defaults
conda config --add channels conda-forge
conda config --set show_channel_urls true
- conda install --yes --quiet conda-forge-build-setup
- source run_conda_forge_build_setup
+ conda install --yes --quiet conda-build
script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- - git -C arrow checkout {{ ARROW_SHA }}
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
+ - git -C arrow checkout {{ arrow.head }}
- pushd arrow/dev/tasks/conda-recipes
- - conda build --output-folder . parquet-cpp arrow-cpp pyarrow
+ - conda build --output-folder . -m $CONFIG parquet-cpp arrow-cpp pyarrow
+ - pushd linux-64
- |
- pushd linux-64
for file in *.tar.bz2; do
mv "$file" "$(basename "$file" .tar.bz2)-linux-64.tar.bz2"
done
@@ -75,4 +75,4 @@ deploy:
notifications:
email:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/conda-recipes/travis.osx.yml b/dev/tasks/conda-recipes/travis.osx.yml
index 31c1c24..dc94668 100644
--- a/dev/tasks/conda-recipes/travis.osx.yml
+++ b/dev/tasks/conda-recipes/travis.osx.yml
@@ -22,12 +22,13 @@ osx_image: xcode6.4
env:
matrix:
- - CONDA_PY=27
- - CONDA_PY=35
- - CONDA_PY=36
+ - CONFIG=variants/osx_python2.7.yaml
+ - CONFIG=variants/osx_python3.5.yaml
+ - CONFIG=variants/osx_python3.6.yaml
global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - ARROW_VERSION={{ ARROW_VERSION }}
+ - TRAVIS_TAG={{ job.branch }}
+ - ARROW_VERSION={{ arrow.version }}
+ - PYTHONUNBUFFERED=1
before_install:
# Remove homebrew.
@@ -58,16 +59,18 @@ install:
conda config --add channels defaults
conda config --add channels conda-forge
conda config --set show_channel_urls true
- conda install --yes --quiet conda-forge-build-setup
+ conda install --yes --quiet conda-forge-ci-setup=1
source run_conda_forge_build_setup
+# conda install --yes --quiet conda-build
+
script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- - git -C arrow checkout {{ ARROW_SHA }}
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
+ - git -C arrow checkout {{ arrow.head }}
- pushd arrow/dev/tasks/conda-recipes
- - conda build --output-folder . parquet-cpp arrow-cpp pyarrow
+ - conda build --output-folder . -m $CONFIG parquet-cpp arrow-cpp pyarrow
+ - pushd osx-64
- |
- pushd osx-64
for file in *.tar.bz2; do
mv "$file" "$(basename "$file" .tar.bz2)-osx-64.tar.bz2"
done
@@ -84,4 +87,4 @@ deploy:
notifications:
email:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/linux_python2.7.yaml
similarity index 68%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/linux_python2.7.yaml
index d9c69e9..45026b0 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/linux_python2.7.yaml
@@ -15,25 +15,33 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '2.7'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/linux_python3.5.yaml
similarity index 68%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/linux_python3.5.yaml
index d9c69e9..683022f 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/linux_python3.5.yaml
@@ -15,25 +15,33 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.5'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/linux_python3.6.yaml
similarity index 68%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/linux_python3.6.yaml
index d9c69e9..6b7d889 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/linux_python3.6.yaml
@@ -15,25 +15,33 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.6'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/osx_python2.7.yaml
similarity index 63%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/osx_python2.7.yaml
index d9c69e9..b8fc15f 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/osx_python2.7.yaml
@@ -15,25 +15,39 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+MACOSX_DEPLOYMENT_TARGET:
+- '10.9'
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+macos_machine:
+- x86_64-apple-darwin13.4.0
+macos_min_version:
+- '10.9'
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '2.7'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/osx_python3.5.yaml
similarity index 63%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/osx_python3.5.yaml
index d9c69e9..05f7a8d 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/osx_python3.5.yaml
@@ -15,25 +15,39 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+MACOSX_DEPLOYMENT_TARGET:
+- '10.9'
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+macos_machine:
+- x86_64-apple-darwin13.4.0
+macos_min_version:
+- '10.9'
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.5'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/osx_python3.6.yaml
similarity index 68%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/osx_python3.6.yaml
index d9c69e9..6b7d889 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/osx_python3.6.yaml
@@ -15,25 +15,33 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- toolchain_c
+cxx_compiler:
+- toolchain_cxx
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.6'
+snappy:
+- 1.1.7
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.5.yaml
similarity index 66%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.5.yaml
index d9c69e9..d886b0e 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.5.yaml
@@ -15,25 +15,37 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- vs2015
+cxx_compiler:
+- vs2015
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.5'
+snappy:
+- 1.1.7
+zip_keys:
+- - python
+ - c_compiler
+ - cxx_compiler
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/.gitignore b/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
similarity index 66%
copy from .gitignore
copy to dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
index d9c69e9..880642f 100644
--- a/.gitignore
+++ b/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
@@ -15,25 +15,37 @@
# specific language governing permissions and limitations
# under the License.
-# Compiled source
-*.a
-*.dll
-*.o
-*.py[ocd]
-*.so
-*.so.*
-*.dylib
-.build_cache_dir
-MANIFEST
-
-# Generated Visual Studio files
-*.vcxproj
-*.vcxproj.*
-*.sln
-*.iml
-
-cpp/.idea/
-python/.eggs/
-.vscode
-.idea/
-.pytest_cache/
+boost_cpp:
+- 1.67.0
+c_compiler:
+- vs2015
+cxx_compiler:
+- vs2015
+lz4_c:
+- 1.8.1
+pin_run_as_build:
+ boost-cpp:
+ max_pin: x.x.x
+ lz4-c:
+ max_pin: x.x.x
+ python:
+ min_pin: x.x
+ max_pin: x.x
+ snappy:
+ max_pin: x.x.x
+ zlib:
+ max_pin: x.x
+ zstd:
+ max_pin: x.x.x
+python:
+- '3.6'
+snappy:
+- 1.1.7
+zip_keys:
+- - python
+ - c_compiler
+ - cxx_compiler
+zlib:
+- '1.2'
+zstd:
+- 1.3.3
diff --git a/dev/tasks/config-nightlies/travis.linux.yml b/dev/tasks/config-nightlies/travis.linux.yml
deleted file mode 100644
index bf2774b..0000000
--- a/dev/tasks/config-nightlies/travis.linux.yml
+++ /dev/null
@@ -1,67 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-branches:
- # don't attempt to build branches intented for windows builds
- except:
- - /.*win.*/
-
-os: linux
-dist: trusty
-language: generic
-
-before_install:
- # Install Miniconda.
- - echo `pwd`
- - |
- echo ""
- echo "Installing a fresh version of Miniconda."
- MINICONDA_URL="https://repo.continuum.io/miniconda"
- MINICONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
- curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
- bash $MINICONDA_FILE -b
-
- # Configure conda.
- - |
- echo ""
- echo "Configuring conda."
- source /home/travis/miniconda3/bin/activate root
- conda config --remove channels defaults
- conda config --add channels defaults
- conda config --add channels conda-forge
- conda config --set show_channel_urls true
-
-install:
- - conda install -y -q jinja2 pygit2 click pyyaml setuptools_scm
-
-script:
- # fetch all branches of crossbow
- - git config remote.origin.fetch +refs/heads/*:refs/remotes/origin/*
-
- # clone arrow with crossbow tool
- - pushd ..
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }}
-
- # submit packaging tasks
- - |
- python arrow/dev/tasks/crossbow.py \
- conda-linux \
- conda-win \
- conda-osx \
- wheel-linux \
- wheel-win \
- wheel-osx \
- linux-packages
diff --git a/dev/tasks/crossbow.py b/dev/tasks/crossbow.py
index a2a29eb..3b71358 100755
--- a/dev/tasks/crossbow.py
+++ b/dev/tasks/crossbow.py
@@ -19,18 +19,22 @@
import os
import re
-import yaml
+import sys
import time
import click
import pygit2
import github3
-import logging
-from enum import Enum
+from io import StringIO
from pathlib import Path
+from datetime import date
from textwrap import dedent
-from jinja2 import Template
+from jinja2 import Template, StrictUndefined
from setuptools_scm import get_version
+from ruamel.yaml import YAML
+
+
+CWD = Path(__file__).parent.absolute()
class GitRemoteCallbacks(pygit2.RemoteCallbacks):
@@ -63,10 +67,18 @@ class GitRemoteCallbacks(pygit2.RemoteCallbacks):
class Repo(object):
+ """Base class for interaction with local git repositories
- def __init__(self, repo_path):
- self.path = Path(repo_path).absolute()
+ A high level wrapper used for both reading revision information from
+ arrow's repository and pushing continuous integration tasks to the queue
+ repository.
+ """
+
+ def __init__(self, path, github_token=None):
+ self.path = Path(path)
self.repo = pygit2.Repository(str(self.path))
+ self.github_token = github_token
+ self._updated_refs = []
def __str__(self):
tpl = dedent('''
@@ -79,29 +91,33 @@ class Repo(object):
head=self.head
)
+ @property
+ def origin(self):
+ return self.repo.remotes['origin']
+
def fetch(self):
- self.origin.fetch()
+ refspec = '+refs/heads/*:refs/remotes/origin/*'
+ self.origin.fetch([refspec])
+
+ def push(self):
+ callbacks = GitRemoteCallbacks(self.github_token)
+ self.origin.push(self._updated_refs, callbacks=callbacks)
+ self.updated_refs = []
@property
def head(self):
"""Currently checked out commit's sha"""
- return self.repo.head.target
+ return self.repo.head
@property
def branch(self):
"""Currently checked out branch"""
- reference = self.repo.head.shorthand
- return self.repo.branches[reference]
+ return self.repo.branches[self.repo.head.shorthand]
@property
def remote(self):
"""Currently checked out branch's remote counterpart"""
- remote_name = self.branch.upstream.remote_name
- return self.repo.remotes[remote_name]
-
- @property
- def origin(self):
- return self.repo.remotes['origin']
+ return self.repo.remotes[self.branch.upstream.remote_name]
@property
def email(self):
@@ -112,29 +128,7 @@ class Repo(object):
name = next(self.repo.config.get_multivar('user.name'))
return pygit2.Signature(name, self.email, int(time.time()))
- def parse_user_repo(self):
- m = re.match('.*\/([^\/]+)\/([^\/\.]+)(\.git)?$', self.remote.url)
- user, repo = m.group(1), m.group(2)
- return user, repo
-
-
-class Queue(Repo):
-
- def __init__(self, repo_path):
- super(Queue, self).__init__(repo_path)
- self._updated_refs = []
-
- def next_job_id(self, prefix):
- """Auto increments the branch's identifier based on the prefix"""
- pattern = re.compile(prefix + '-(\d+)')
- matches = list(filter(None, map(pattern.match, self.repo.branches)))
- if matches:
- latest = max(int(m.group(1)) for m in matches)
- else:
- latest = 0
- return '{}-{}'.format(prefix, latest + 1)
-
- def _create_branch(self, branch_name, files, parents=[], message=''):
+ def create_branch(self, branch_name, files, parents=[], message=''):
# 1. create tree
builder = self.repo.TreeBuilder()
@@ -153,12 +147,13 @@ class Queue(Repo):
# 3. create branch pointing to the previously created commit
branch = self.repo.create_branch(branch_name, commit)
+
# append to the pushable references
self._updated_refs.append('refs/heads/{}'.format(branch_name))
return branch
- def _create_tag(self, tag_name, commit_id, message=''):
+ def create_tag(self, tag_name, commit_id, message=''):
tag_id = self.repo.create_tag(tag_name, commit_id,
pygit2.GIT_OBJ_COMMIT, self.signature,
message)
@@ -168,36 +163,146 @@ class Queue(Repo):
return self.repo[tag_id]
+ def file_contents(self, commit_id, file):
+ commit = self.repo[commit_id]
+ entry = commit.tree[file]
+ blob = self.repo[entry.id]
+ return blob.data
+
+ def _parse_github_user_repo(self):
+ m = re.match('.*\/([^\/]+)\/([^\/\.]+)(\.git)?$', self.remote.url)
+ user, repo = m.group(1), m.group(2)
+ return user, repo
+
+ def as_github_repo(self):
+ """Converts it to a repository object which wraps the GitHub API"""
+ username, reponame = self._parse_github_user_repo()
+ gh = github3.login(token=self.github_token)
+ return gh.repository(username, reponame)
+
+
+class Queue(Repo):
+
+ def _next_job_id(self, prefix):
+ """Auto increments the branch's identifier based on the prefix"""
+ pattern = re.compile('[\w\/-]*{}-(\d+)'.format(prefix))
+ matches = list(filter(None, map(pattern.match, self.repo.branches)))
+ if matches:
+ latest = max(int(m.group(1)) for m in matches)
+ else:
+ latest = 0
+ return '{}-{}'.format(prefix, latest + 1)
+
+ def get(self, job_name):
+ branch_name = 'origin/{}'.format(job_name)
+ branch = self.repo.branches[branch_name]
+ content = self.file_contents(branch.target, 'job.yml')
+ buffer = StringIO(content.decode('utf-8'))
+ return yaml.load(buffer)
+
def put(self, job, prefix='build'):
assert isinstance(job, Job)
- assert job.branch is not None
+ assert job.branch is None
+
+ # auto increment and set next job id, e.g. build-85
+ job.branch = self._next_job_id(prefix)
# create tasks' branches
for task_name, task in job.tasks.items():
- branch = self._create_branch(task.branch, files=task.files())
+ task.branch = '{}-{}'.format(job.branch, task_name)
+ files = task.render_files(job=job, arrow=job.target)
+ branch = self.create_branch(task.branch, files=files)
task.commit = str(branch.target)
# create job's branch
- branch = self._create_branch(job.branch, files=job.files())
- self._create_tag(job.branch, branch.target)
+ branch = self.create_branch(job.branch, files=job.render_files())
+ self.create_tag(job.branch, branch.target)
return branch
- def push(self, token):
- callbacks = GitRemoteCallbacks(token)
- self.origin.push(self._updated_refs, callbacks=callbacks)
- self.updated_refs = []
+ def github_statuses(self, job):
+ repo = self.as_github_repo()
+ return {name: repo.commit(task.commit).status()
+ for name, task in job.tasks.items()}
+
+ def github_assets(self, job):
+ repo = self.as_github_repo()
+ try:
+ release = repo.release_from_tag(job.branch)
+ except github3.exceptions.NotFoundError:
+ return {}
+ else:
+ return {a.name: a for a in release.assets()}
+
+ def upload_assets(self, job, files, content_type):
+ repo = self.as_github_repo()
+ release = repo.release_from_tag(job.branch)
+ assets = {a.name: a for a in release.assets()}
+
+ for path in files:
+ if path.name in assets:
+ # remove already uploaded asset
+ assets[path.name].delete()
+ with path.open('rb') as fp:
+ release.upload_asset(name=path.name, asset=fp,
+ content_type=content_type)
+
+
+class Target(object):
+ """Describes target repository and revision the builds run against
+ This serializable data container holding information about arrow's
+ git remote, branch, sha and version number as well as some metadata
+ (currently only an email address where the notification should be sent).
+ """
+
+ def __init__(self, head, branch, remote, version, email=None):
+ self.head = head
+ self.email = email
+ self.branch = branch
+ self.remote = remote
+ self.version = version
+
+ @classmethod
+ def from_repo(cls, repo_path):
+ repo = Repo(repo_path)
+ version = get_version(repo.path, local_scheme=lambda v: '')
+ return cls(head=str(repo.head.target),
+ email=repo.email,
+ branch=repo.branch.branch_name,
+ remote=repo.remote.url,
+ version=version)
+
+
+class Task(object):
+ """Describes a build task and metadata required to render CI templates
-class Platform(Enum):
- # in alphabetical order
- LINUX = 0
- OSX = 1
- WIN = 2
+ A task is represented as a single git commit and branch containing jinja2
+ rendered files (currently appveyor.yml or .travis.yml configurations).
+
+ A task can't be directly submitted to a queue, must belong to a job.
+ Each task's unique identifier is its branch name, which is generated after
+ submitting the job to a queue.
+ """
+
+ def __init__(self, platform, template, artifacts=None, params=None):
+ assert platform in {'win', 'osx', 'linux'}
+ self.platform = platform
+ self.template = template
+ self.artifacts = artifacts or []
+ self.params = params or {}
+ self.branch = None # filled after adding to a queue
+ self.commit = None
+
+ def render_files(self, **extra_params):
+ path = CWD / self.template
+ template = Template(path.read_text(), undefined=StrictUndefined)
+ rendered = template.render(**self.params, **extra_params)
+ return {self.filename: rendered}
@property
def ci(self):
- if self is self.WIN:
+ if self.platform == 'win':
return 'appveyor'
else:
return 'travis'
@@ -210,68 +315,45 @@ class Platform(Enum):
return '.travis.yml'
-class Task(object):
-
- def __init__(self, platform, template, commit=None, branch=None, **params):
- assert isinstance(platform, Platform)
- assert isinstance(template, Path)
- self.platform = platform
- self.template = template
- self.branch = branch
- self.commit = commit
- self.params = params
-
- def to_dict(self):
- return {'branch': self.branch,
- 'commit': str(self.commit),
- 'platform': self.platform.name,
- 'template': str(self.template),
- 'params': self.params}
-
- @classmethod
- def from_dict(cls, data):
- return Task(platform=Platform[data['platform'].upper()],
- template=Path(data['template']),
- commit=data.get('commit'),
- branch=data.get('branch'),
- **data.get('params', {}))
-
- def files(self):
- template = Template(self.template.read_text())
- rendered = template.render(**self.params)
- return {self.platform.filename: rendered}
-
-
class Job(object):
+ """Describes multiple tasks against a single target repository"""
- def __init__(self, tasks, branch=None):
+ def __init__(self, target, tasks):
+ assert isinstance(target, Target)
assert all(isinstance(task, Task) for task in tasks.values())
- self.branch = branch
+ self.target = target
self.tasks = tasks
+ self.branch = None # filled after adding to a queue
- def to_dict(self):
- tasks = {name: task.to_dict() for name, task in self.tasks.items()}
- return {'branch': self.branch,
- 'tasks': tasks}
+ def render_files(self):
+ with StringIO() as buf:
+ yaml.dump(self, buf)
+ content = buf.getvalue()
+ return {'job.yml': content}
- @classmethod
- def from_dict(cls, data):
- tasks = {name: Task.from_dict(task)
- for name, task in data['tasks'].items()}
- return Job(tasks=tasks, branch=data.get('branch'))
-
- def files(self):
- return {'job.yml': yaml.dump(self.to_dict(), default_flow_style=False)}
+ @property
+ def email(self):
+ return os.environ.get('CROSSBOW_EMAIL', self.target.email)
-# this should be the mailing list
-MESSAGE_EMAIL = 'szucs.krisztian@gmail.com'
+# configure yaml serializer
+yaml = YAML()
+yaml.register_class(Job)
+yaml.register_class(Task)
+yaml.register_class(Target)
-CWD = Path(__file__).absolute()
+# state color mapping to highlight console output
+COLORS = {'ok': 'green',
+ 'error': 'red',
+ 'missing': 'red',
+ 'failure': 'red',
+ 'pending': 'yellow',
+ 'success': 'green'}
-DEFAULT_CONFIG_PATH = CWD.parent / 'tasks.yml'
-DEFAULT_ARROW_PATH = CWD.parents[2]
-DEFAULT_QUEUE_PATH = CWD.parents[3] / 'crossbow'
+# define default paths
+DEFAULT_CONFIG_PATH = CWD / 'tasks.yml'
+DEFAULT_ARROW_PATH = CWD.parents[1]
+DEFAULT_QUEUE_PATH = CWD.parents[2] / 'crossbow'
@click.group()
@@ -334,53 +416,43 @@ def config_path_validation_callback(ctx, param, value):
@github_token
def submit(task_names, job_prefix, config_path, dry_run, arrow_path,
queue_path, github_token):
- target = Repo(arrow_path)
- queue = Queue(queue_path)
-
- logging.info(target)
- logging.info(queue)
-
- queue.fetch()
-
- version = get_version(arrow_path, local_scheme=lambda v: '')
- job_id = queue.next_job_id(prefix=job_prefix)
-
- variables = {
- # these should be renamed
- 'PLAT': 'x86_64',
- 'EMAIL': os.environ.get('CROSSBOW_EMAIL', target.email),
- 'BUILD_TAG': job_id,
- 'BUILD_REF': str(target.head),
- 'ARROW_SHA': str(target.head),
- 'ARROW_REPO': target.remote.url,
- 'ARROW_BRANCH': target.branch.branch_name,
- 'ARROW_VERSION': version,
- 'PYARROW_VERSION': version,
- }
+ queue = Queue(queue_path, github_token=github_token)
+ target = Target.from_repo(arrow_path)
with Path(config_path).open() as fp:
config = yaml.load(fp)
- # create and filter tasks
- tasks = {name: Task.from_dict(task)
- for name, task in config['tasks'].items()}
- tasks = {name: tasks[name] for name in task_names}
-
- for task_name, task in tasks.items():
- task.branch = '{}-{}'.format(job_id, task_name)
- task.params.update(variables)
-
- # create job
- job = Job(tasks)
- job.branch = job_id
-
- yaml_format = yaml.dump(job.to_dict(), default_flow_style=False)
- click.echo(yaml_format.strip())
-
- if not dry_run:
+ today = date.today().strftime('%Y%m%d')
+ tasks = {}
+ for task_name in task_names:
+ task = config['tasks'][task_name]
+ # replace version number and current date in artifact names
+ artifacts = task.pop('artifacts', None) or []
+ artifacts = [fname.format(version=target.version, date=today)
+ for fname in artifacts]
+ # create task instance from configuration
+ tasks[task_name] = Task(**task, artifacts=artifacts)
+
+ # create job instance, doesn't mutate git data yet
+ job = Job(target=target, tasks=tasks)
+
+ if dry_run:
+ yaml.dump(job, sys.stdout)
+ delimiter = '-' * 79
+ for task_name, task in job.tasks.items():
+ files = task.render_files(job=job, arrow=job.target)
+ for filename, content in files.items():
+ click.echo('\n\n')
+ click.echo(delimiter)
+ click.echo('{:<29}{:>50}'.format(task_name, filename))
+ click.echo(delimiter)
+ click.echo(content)
+ else:
+ queue.fetch()
queue.put(job)
- queue.push(token=github_token)
- click.echo('Pushed job identifier is: `{}`'.format(job_id))
+ queue.push()
+ yaml.dump(job, sys.stdout)
+ click.echo('Pushed job identifier is: `{}`'.format(job.branch))
@crossbow.command()
@@ -390,46 +462,90 @@ def submit(task_names, job_prefix, config_path, dry_run, arrow_path,
'Defaults to crossbow directory placed next to arrow')
@github_token
def status(job_name, queue_path, github_token):
- queue = Queue(queue_path)
- username, reponame = queue.parse_user_repo()
-
- gh = github3.login(token=github_token)
- repo = gh.repository(username, reponame)
- content = repo.file_contents('job.yml', job_name)
-
- job = Job.from_dict(yaml.load(content.decoded))
+ queue = Queue(queue_path, github_token=github_token)
+ queue.fetch()
- tpl = '[{:>7}] {:<24} {:<40}'
- header = tpl.format('status', 'branch', 'sha')
+ tpl = '[{:>7}] {:<24} {:>45}'
+ header = tpl.format('status', 'branch', 'artifacts')
click.echo(header)
click.echo('-' * len(header))
- for name, task in job.tasks.items():
- commit = repo.commit(task.commit)
- status = commit.status()
+ job = queue.get(job_name)
+ assets = queue.github_assets(job)
+ statuses = queue.github_statuses(job)
- click.echo(tpl.format(status.state, task.branch, task.commit))
+ for task_name, task in job.tasks.items():
+ status = statuses[task_name]
+
+ uploaded = 'uploaded {} / {}'.format(
+ sum(a in assets for a in task.artifacts),
+ len(task.artifacts)
+ )
+ leadline = tpl.format(status.state.upper(), task.branch, uploaded)
+ click.echo(click.style(leadline, fg=COLORS[status.state]))
+
+ for artifact in task.artifacts:
+ if artifact in assets:
+ state = 'ok'
+ elif status.state == 'pending':
+ state = 'pending'
+ else:
+ state = 'missing'
+
+ filename = '{:>70} '.format(artifact)
+ statemsg = '[{:>7}]'.format(state.upper())
+ click.echo(filename + click.style(statemsg, fg=COLORS[state]))
@crossbow.command()
@click.argument('job-name', required=True)
-@click.option('--target-dir', default=DEFAULT_ARROW_PATH,
+@click.option('--gpg-homedir', default=None,
+ help=('Full pathname to directory containing the public and '
+ 'private keyrings. Default is whatever GnuPG defaults to'))
+@click.option('--target-dir', default=DEFAULT_ARROW_PATH / 'pkgs',
help='Directory to download the build artifacts')
@click.option('--queue-path', default=DEFAULT_QUEUE_PATH,
help='The repository path used for scheduling the tasks. '
'Defaults to crossbow directory placed next to arrow')
@github_token
-def artifacts(job_name, target_dir, queue_path, github_token):
- queue = Queue(queue_path)
- username, reponame = queue.parse_user_repo()
+def sign(job_name, gpg_homedir, target_dir, queue_path, github_token):
+ """Download and sign build artifacts from github releases"""
+
+ import gnupg
+ gpg = gnupg.GPG(gnupghome=gpg_homedir)
- gh = github3.login(token=github_token)
- repo = gh.repository(username, reponame)
- release = repo.release_from_tag(job_name)
+ # initialize and fetch the queue repository
+ queue = Queue(queue_path, github_token=github_token)
+ queue.fetch()
- for asset in release.assets():
- click.echo('Downloading asset {} ...'.format(asset.name))
- asset.download(target_dir / asset.name)
+ # query the job's artifacts
+ job = queue.get(job_name)
+ assets = queue.github_assets(job)
+
+ click.echo('Downloading and signing assets...')
+ target_dir = Path(target_dir).absolute()
+ target_dir.mkdir(exist_ok=True)
+
+ tpl = '[{:>8}] '
+ for task_name, task in job.tasks.items():
+ for artifact in task.artifacts:
+ if artifact not in assets:
+ msg = click.style(tpl.format('MISSING'), fg=COLORS['missing'])
+ click.echo(msg + artifact)
+ continue
+
+ # download artifact
+ target_path = target_dir / artifact
+ assets[artifact].download(target_path)
+
+ # sign the artifact
+ with target_path.open('rb') as fp:
+ signature_path = Path(str(target_path) + '.sig')
+ gpg.sign_file(fp, detach=True, clearsign=False,
+ output=str(signature_path))
+
+ msg = click.style(tpl.format('SIGNED'), fg=COLORS['ok'])
+ click.echo(msg + artifact)
if __name__ == '__main__':
diff --git a/dev/tasks/linux-packages/Rakefile b/dev/tasks/linux-packages/Rakefile
index 1dec2d7..0bcf5b5 100644
--- a/dev/tasks/linux-packages/Rakefile
+++ b/dev/tasks/linux-packages/Rakefile
@@ -34,6 +34,7 @@ class ApacheArrowPackageTask < PackageTask
def detect_version(release_time)
pom_xml_path = File.join(arrow_source_dir, "java", "pom.xml")
version = File.read(pom_xml_path).scan(/^ <version>(.+?)<\/version>/)[0][0]
+ version = ENV['ARROW_VERSION'] or version #try to read from env
formatted_release_time = release_time.strftime("%Y%m%d")
version.gsub(/-SNAPSHOT\z/) {".#{formatted_release_time}"}
end
diff --git a/dev/tasks/conda-recipes/arrow-cpp/build.sh b/dev/tasks/linux-packages/apt/ubuntu-bionic/Dockerfile
similarity index 56%
copy from dev/tasks/conda-recipes/arrow-cpp/build.sh
copy to dev/tasks/linux-packages/apt/ubuntu-bionic/Dockerfile
index 7ec43cc..e4362b7 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/build.sh
+++ b/dev/tasks/linux-packages/apt/ubuntu-bionic/Dockerfile
@@ -1,5 +1,3 @@
-#!/bin/sh
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,29 +15,33 @@
# specific language governing permissions and limitations
# under the License.
-set -e
-set -x
-
-# Build dependencies
-export ARROW_BUILD_TOOLCHAIN=$PREFIX
+FROM ubuntu:18.04
-cd cpp
-mkdir build-dir
-cd build-dir
+ENV DEBIAN_FRONTEND noninteractive
-cmake \
- -DCMAKE_BUILD_TYPE=release \
- -DCMAKE_INSTALL_PREFIX=$PREFIX \
- -DCMAKE_INSTALL_LIBDIR=$PREFIX/lib \
- -DARROW_BOOST_USE_SHARED=ON \
- -DARROW_BUILD_BENCHMARKS=OFF \
- -DARROW_BUILD_UTILITIES=OFF \
- -DARROW_BUILD_TESTS=OFF \
- -DARROW_JEMALLOC=OFF \
- -DARROW_PLASMA=ON \
- -DARROW_PYTHON=ON \
- -DARROW_ORC=ON \
- ..
+ARG DEBUG
-make -j${CPU_COUNT}
-make install
+RUN \
+ quiet=$([ "${DEBUG}" = "yes" ] || echo "-qq") && \
+ apt update ${quiet} && \
+ apt install -y -V ${quiet} \
+ autoconf-archive \
+ build-essential \
+ cmake \
+ debhelper\
+ devscripts \
+ git \
+ gtk-doc-tools \
+ libboost-filesystem-dev \
+ libboost-regex-dev \
+ libboost-system-dev \
+ libgirepository1.0-dev \
+ libglib2.0-doc \
+ libjemalloc-dev \
+ lsb-release \
+ nvidia-cuda-toolkit \
+ pkg-config \
+ python3-dev \
+ python3-numpy && \
+ apt clean && \
+ rm -rf /var/lib/apt/lists/*
diff --git a/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install b/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
index 62de6ce..18ea2e3 100644
--- a/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
+++ b/dev/tasks/linux-packages/debian.ubuntu-trusty/libarrow-dev.install
@@ -3,3 +3,4 @@ usr/lib/*/libarrow.a
usr/lib/*/libarrow.so
usr/lib/*/pkgconfig/arrow.pc
usr/lib/*/pkgconfig/arrow-compute.pc
+usr/lib/*/pkgconfig/arrow-orc.pc
diff --git a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
index 783b7a7..b1085d1 100755
--- a/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
+++ b/dev/tasks/linux-packages/debian.ubuntu-trusty/rules
@@ -6,6 +6,8 @@
# This has to be exported to make some magic below work.
export DH_OPTIONS
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
BUILD_TYPE=release
%:
diff --git a/dev/tasks/linux-packages/debian/libarrow-dev.install b/dev/tasks/linux-packages/debian/libarrow-dev.install
index 62de6ce..18ea2e3 100644
--- a/dev/tasks/linux-packages/debian/libarrow-dev.install
+++ b/dev/tasks/linux-packages/debian/libarrow-dev.install
@@ -3,3 +3,4 @@ usr/lib/*/libarrow.a
usr/lib/*/libarrow.so
usr/lib/*/pkgconfig/arrow.pc
usr/lib/*/pkgconfig/arrow-compute.pc
+usr/lib/*/pkgconfig/arrow-orc.pc
diff --git a/dev/tasks/linux-packages/debian/rules b/dev/tasks/linux-packages/debian/rules
index 5a3c0a3..0659bd8 100755
--- a/dev/tasks/linux-packages/debian/rules
+++ b/dev/tasks/linux-packages/debian/rules
@@ -6,10 +6,12 @@
# This has to be exported to make some magic below work.
export DH_OPTIONS
+export DEB_BUILD_MAINT_OPTIONS=reproducible=-timeless
+
BUILD_TYPE=release
%:
- dh $@ --with gir,autoreconf
+ dh $@ --with gir,autoreconf --no-parallel
override_dh_autoreconf:
dh_autoreconf \
diff --git a/dev/tasks/linux-packages/package-task.rb b/dev/tasks/linux-packages/package-task.rb
index bb01a02..0c7d768 100644
--- a/dev/tasks/linux-packages/package-task.rb
+++ b/dev/tasks/linux-packages/package-task.rb
@@ -211,6 +211,7 @@ VERSION=#{@version}
"ubuntu-trusty",
"ubuntu-xenial",
"ubuntu-artful",
+ "ubuntu-bionic",
]
end
targets.each do |target|
diff --git a/dev/tasks/linux-packages/travis.linux.yml b/dev/tasks/linux-packages/travis.linux.yml
index 8cc6377..b151a30 100644
--- a/dev/tasks/linux-packages/travis.linux.yml
+++ b/dev/tasks/linux-packages/travis.linux.yml
@@ -22,19 +22,55 @@ language: ruby
env:
global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - PLAT={{ PLAT }}
- - BUILD_REF={{ BUILD_REF }}
- - PYARROW_VERSION={{ PYARROW_VERSION }}
+ - PLAT=x86_64
+ - TRAVIS_TAG={{ job.branch }}
+ - BUILD_REF={{ arrow.head }}
+ - ARROW_VERSION={{ arrow.version }}
matrix:
include:
- script:
- - (cd arrow/dev/tasks/linux-packages && travis_wait 40 && rake version:update && rake apt:build APT_TARGETS=debian-stretch,ubuntu-trusty,ubuntu-xenial PARALLEL=yes DEBUG=no)
+ - pushd arrow/dev/tasks/linux-packages
+ - |
+ rake version:update && travis_wait 60 rake apt:build \
+ APT_TARGETS=debian-stretch,ubuntu-trusty,ubuntu-xenial \
+ PARALLEL=yes \
+ DEBUG=no
+ deploy:
+ provider: releases
+ api_key: $CROSSBOW_GITHUB_TOKEN
+ file_glob: true
+ file: "**/*.deb"
+ skip_cleanup: true
+ on:
+ tags: true
- script:
- - (cd arrow/dev/tasks/linux-packages && travis_wait 40 && rake version:update && rake apt:build APT_TARGETS=ubuntu-artful PARALLEL=yes DEBUG=no)
+ - pushd arrow/dev/tasks/linux-packages
+ - |
+ rake version:update && travis_wait 60 rake apt:build \
+ APT_TARGETS=ubuntu-artful,ubuntu-bionic \
+ PARALLEL=yes \
+ DEBUG=no
+ # TODO(kszucs) resolve artifact naming conflict with other debian pkgs
+ # deploy:
+ # provider: releases
+ # api_key: $CROSSBOW_GITHUB_TOKEN
+ # file_glob: true
+ # file: "**/*.deb"
+ # skip_cleanup: true
+ # on:
+ # tags: true
- script:
- - (cd arrow/dev/tasks/linux-packages && rake version:update && rake yum:build PARALLEL=yes DEBUG=no)
+ - pushd arrow/dev/tasks/linux-packages
+ - rake version:update && rake yum:build PARALLEL=yes DEBUG=no
+ deploy:
+ provider: releases
+ api_key: $CROSSBOW_GITHUB_TOKEN
+ file_glob: true
+ file: "**/*.rpm"
+ skip_cleanup: true
+ on:
+ tags: true
before_install:
- sudo apt update -y -qq
@@ -60,18 +96,9 @@ before_install:
python3-numpy
before_script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- - git -C arrow checkout {{ ARROW_SHA }}
-
-deploy:
- provider: releases
- api_key: $CROSSBOW_GITHUB_TOKEN
- file_glob: true
- file: /path/to/pachages/*.tar.gz # FIXME(kszucs) after the builds pass
- skip_cleanup: true
- on:
- tags: true
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
+ - git -C arrow checkout {{ arrow.head }}
notifications:
email:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/linux-packages/yum/arrow.spec.in b/dev/tasks/linux-packages/yum/arrow.spec.in
index 65c357e..76697f4 100644
--- a/dev/tasks/linux-packages/yum/arrow.spec.in
+++ b/dev/tasks/linux-packages/yum/arrow.spec.in
@@ -132,6 +132,7 @@ Libraries and header files for Apache Arrow C++.
%{_libdir}/libarrow.so
%{_libdir}/pkgconfig/arrow.pc
%{_libdir}/pkgconfig/arrow-compute.pc
+%{_libdir}/pkgconfig/arrow-orc.pc
%if %{use_python}
%package python-libs
diff --git a/dev/tasks/conda-recipes/travis.linux.yml b/dev/tasks/nightlies.sample.yml
similarity index 54%
copy from dev/tasks/conda-recipes/travis.linux.yml
copy to dev/tasks/nightlies.sample.yml
index 84ca83b..7205bce 100644
--- a/dev/tasks/conda-recipes/travis.linux.yml
+++ b/dev/tasks/nightlies.sample.yml
@@ -15,21 +15,22 @@
# specific language governing permissions and limitations
# under the License.
-language: generic
+# this travis configuration can be used to submit cron scheduled tasks
+# 1. copy this file to one of crossbow's branch (master for example) with
+# filename .travis.yml
+# 2. setup daily cron jobs for that particular branch, see travis'
+# documentation https://docs.travis-ci.com/user/cron-jobs/
+
+branches:
+ # don't attempt to build branches intented for windows builds
+ except:
+ - /.*win.*/
os: linux
dist: trusty
+language: generic
-env:
- matrix:
- - CONDA_PY=27
- - CONDA_PY=35
- - CONDA_PY=36
- global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - ARROW_VERSION={{ ARROW_VERSION }}
-
-install:
+before_install:
# Install Miniconda.
- echo `pwd`
- |
@@ -49,30 +50,42 @@ install:
conda config --add channels defaults
conda config --add channels conda-forge
conda config --set show_channel_urls true
- conda install --yes --quiet conda-forge-build-setup
- source run_conda_forge_build_setup
-script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- - git -C arrow checkout {{ ARROW_SHA }}
- - pushd arrow/dev/tasks/conda-recipes
- - conda build --output-folder . parquet-cpp arrow-cpp pyarrow
+install:
- |
- pushd linux-64
- for file in *.tar.bz2; do
- mv "$file" "$(basename "$file" .tar.bz2)-linux-64.tar.bz2"
- done
- popd
+ conda install -y \
+ jinja2 \
+ pygit2 \
+ click \
+ ruamel.yaml \
+ setuptools_scm \
+ github3.py \
+ python-gnupg
-deploy:
- provider: releases
- api_key: $CROSSBOW_GITHUB_TOKEN
- file_glob: true
- file: $TRAVIS_BUILD_DIR/arrow/dev/tasks/conda-recipes/linux-64/*.tar.bz2
- skip_cleanup: true
- on:
- tags: true
+script:
+ # to build against a specific branch of a fork
+ # git clone -b <branch> https://github.com/<user>/arrow
+ - pushd ..
+ - git clone -b master https://github.com/apache/arrow
-notifications:
- email:
- - {{ EMAIL }}
+ # submit packaging tasks
+ - |
+ if [ $TRAVIS_EVENT_TYPE = "cron" ]; then
+ python arrow/dev/tasks/crossbow.py submit \
+ conda-linux \
+ conda-win \
+ conda-osx \
+ wheel-linux \
+ wheel-win \
+ wheel-osx \
+ linux-packages
+ else
+ python arrow/dev/tasks/crossbow.py submit --dry-run \
+ conda-linux \
+ conda-win \
+ conda-osx \
+ wheel-linux \
+ wheel-win \
+ wheel-osx \
+ linux-packages
+ fi
diff --git a/dev/tasks/python-wheels/appveyor.yml b/dev/tasks/python-wheels/appveyor.yml
index b3407fb..83ebe60 100644
--- a/dev/tasks/python-wheels/appveyor.yml
+++ b/dev/tasks/python-wheels/appveyor.yml
@@ -30,8 +30,8 @@ environment:
MSVC_DEFAULT_OPTIONS: ON
BOOST_ROOT: C:\Libraries\boost_1_63_0
BOOST_LIBRARYDIR: C:\Libraries\boost_1_63_0\lib64-msvc-14.0
- pyarrow_version: {{ PYARROW_VERSION }}
- pyarrow_ref: {{ BUILD_REF }}
+ pyarrow_version: {{ arrow.version }}
+ pyarrow_ref: {{ arrow.head }}
parquet_cpp_ref: apache-parquet-cpp-1.4.0
ARROW_SRC: C:\apache-arrow
@@ -41,8 +41,8 @@ init:
build_script:
- mkdir wheels
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} %ARROW_SRC% || exit /B
- - git -C arrow checkout {{ ARROW_SHA }} || exit /B
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} %ARROW_SRC% || exit /B
+ - git -C arrow checkout {{ arrow.head }} || exit /B
- call %ARROW_SRC%\dev\tasks\python-wheels\win-build.bat
after_build:
@@ -52,7 +52,7 @@ artifacts:
- path: wheels\*.whl
deploy:
- release: {{ BUILD_TAG }}
+ release: {{ job.branch }}
provider: GitHub
auth_token: "%CROSSBOW_GITHUB_TOKEN%"
artifact: /.*\.whl/
@@ -62,4 +62,4 @@ deploy:
notifications:
- provider: Email
to:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/python-wheels/travis.linux.yml b/dev/tasks/python-wheels/travis.linux.yml
index 2685ad4..3f4e6e4 100644
--- a/dev/tasks/python-wheels/travis.linux.yml
+++ b/dev/tasks/python-wheels/travis.linux.yml
@@ -16,10 +16,10 @@
env:
global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - PLAT={{ PLAT }}
- - BUILD_REF={{ BUILD_REF }}
- - PYARROW_VERSION={{ PYARROW_VERSION }}
+ - PLAT=x86_64
+ - TRAVIS_TAG={{ job.branch }}
+ - BUILD_REF={{ arrow.head }}
+ - PYARROW_VERSION={{ arrow.version }}
os: linux
dist: trusty
@@ -32,8 +32,8 @@ before_script:
- docker pull quay.io/xhochy/arrow_manylinux1_x86_64_base:latest
script:
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
- - git -C arrow checkout {{ ARROW_SHA }}
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
+ - git -C arrow checkout {{ arrow.head }}
- mkdir -p dist
- pushd arrow/python/manylinux1
@@ -57,4 +57,4 @@ deploy:
notifications:
email:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/python-wheels/travis.osx.yml b/dev/tasks/python-wheels/travis.osx.yml
index f346297..10928c4 100644
--- a/dev/tasks/python-wheels/travis.osx.yml
+++ b/dev/tasks/python-wheels/travis.osx.yml
@@ -23,10 +23,10 @@ sudo: required
env:
global:
- - TRAVIS_TAG={{ BUILD_TAG }}
- - PLAT={{ PLAT }}
- - BUILD_REF={{ BUILD_REF }}
- - PYARROW_VERSION={{ PYARROW_VERSION }}
+ - PLAT=x86_64
+ - TRAVIS_TAG={{ job.branch }}
+ - BUILD_REF={{ arrow.head }}
+ - PYARROW_VERSION={{ arrow.version }}
matrix:
exclude:
@@ -50,7 +50,8 @@ matrix:
before_install:
- git clone https://github.com/matthew-brett/multibuild # TODO pin it
- - git clone -b {{ ARROW_BRANCH }} {{ ARROW_REPO }} arrow
+ - git clone -b {{ arrow.branch }} {{ arrow.remote }} arrow
+ - git -C arrow checkout {{ arrow.head }}
# Also remove artifacts that depend on Boost
- brew uninstall boost cgal postgis sfcgal
@@ -84,4 +85,4 @@ deploy:
notifications:
email:
- - {{ EMAIL }}
+ - {{ job.email }}
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 8aaf469..683833e 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -17,38 +17,132 @@
tasks:
# arbitrary_task_name:
- # branch: defaults to name
# platform: osx|linux|win
# template: path of jinja2 templated yml
# params: optional extra parameters
+ # artifacts: list of expected release asset filenames,
+ # version and date variables can be used in the artifact's name, e.g.
+ # - pyarrow-{version}-{date}-py36_0-linux-64.tar.bz2
# conda packages
conda-linux:
platform: linux
template: conda-recipes/travis.linux.yml
+ artifacts:
+ - arrow-cpp-{version}-py27_0-linux-64.tar.bz2
+ - arrow-cpp-{version}-py35_0-linux-64.tar.bz2
+ - arrow-cpp-{version}-py36_0-linux-64.tar.bz2
+ - pyarrow-{version}-py27_0-linux-64.tar.bz2
+ - pyarrow-{version}-py35_0-linux-64.tar.bz2
+ - pyarrow-{version}-py36_0-linux-64.tar.bz2
+ - parquet-cpp-1.4.0-0-linux-64.tar.bz2
conda-osx:
platform: osx
template: conda-recipes/travis.osx.yml
+ artifacts:
+ - arrow-cpp-{version}-py27_0-osx-64.tar.bz2
+ - arrow-cpp-{version}-py35_0-osx-64.tar.bz2
+ - arrow-cpp-{version}-py36_0-osx-64.tar.bz2
+ - pyarrow-{version}-py27_0-osx-64.tar.bz2
+ - pyarrow-{version}-py35_0-osx-64.tar.bz2
+ - pyarrow-{version}-py36_0-osx-64.tar.bz2
+ - parquet-cpp-1.4.0-0-osx-64.tar.bz2
conda-win:
platform: win
template: conda-recipes/appveyor.yml
+ artifacts:
+ - arrow-cpp-{version}-py35_vc14_0.tar.bz2
+ - arrow-cpp-{version}-py36_vc14_0.tar.bz2
+ - pyarrow-{version}-py35_vc14_0.tar.bz2
+ - pyarrow-{version}-py36_vc14_0.tar.bz2
+ - parquet-cpp-1.4.0-vc14_0.tar.bz2
# python wheels
wheel-linux:
platform: linux
template: python-wheels/travis.linux.yml
+ artifacts:
+ - pyarrow-{version}-cp27-cp27m-manylinux1_x86_64.whl
+ - pyarrow-{version}-cp27-cp27mu-manylinux1_x86_64.whl
+ - pyarrow-{version}-cp35-cp35m-manylinux1_x86_64.whl
+ - pyarrow-{version}-cp36-cp36m-manylinux1_x86_64.whl
wheel-osx:
platform: osx
template: python-wheels/travis.osx.yml
+ artifacts:
+ - pyarrow-{version}-cp27-cp27m-macosx_10_6_intel.whl
+ - pyarrow-{version}-cp35-cp35m-macosx_10_6_intel.whl
+ - pyarrow-{version}-cp36-cp36m-macosx_10_6_intel.whl
wheel-win:
platform: win
template: python-wheels/appveyor.yml
+ artifacts:
+ - pyarrow-{version}-cp35-cp35m-win_amd64.whl
+ - pyarrow-{version}-cp36-cp36m-win_amd64.whl
# linux packages
linux-packages:
platform: linux
template: linux-packages/travis.linux.yml
+ artifacts:
+ # DEB: debian-stretch, ubuntu-trusty
+ - gir1.2-arrow-1.0_{version}-1_amd64.deb
+ - gir1.2-arrow-gpu-1.0_{version}-1_amd64.deb
+ - libarrow-dev_{version}-1_amd64.deb
+ - libarrow-glib-dev_{version}-1_amd64.deb
+ - libarrow-glib-doc_{version}-1_all.deb
+ - libarrow-glib0-dbgsym_{version}-1_amd64.deb
+ - libarrow-glib0_{version}-1_amd64.deb
+ - libarrow-gpu-dev_{version}-1_amd64.deb
+ - libarrow-gpu-glib-dev_{version}-1_amd64.deb
+ - libarrow-gpu-glib0-dbgsym_{version}-1_amd64.deb
+ - libarrow-gpu-glib0_{version}-1_amd64.deb
+ - libarrow-gpu0-dbgsym_{version}-1_amd64.deb
+ - libarrow-gpu0_{version}-1_amd64.deb
+ - libarrow-python-dev_{version}-1_amd64.deb
+ - libarrow-python0-dbgsym_{version}-1_amd64.deb
+ - libarrow-python0_{version}-1_amd64.deb
+ - libarrow0-dbgsym_{version}-1_amd64.deb
+ - libarrow0_{version}-1_amd64.deb
+
+ # TODO(kszucs) resolve naming conflict with other deb pkgs
+ # DEB: ubuntu-xenial,ubuntu-artful
+ # - gir1.2-arrow-1.0_{version}-1_amd64.deb
+ # - gir1.2-arrow-gpu-1.0_{version}-1_amd64.deb
+ # - libarrow-dev_{version}-1_amd64.deb
+ # - libarrow-glib-dev_{version}-1_amd64.deb
+ # - libarrow-glib-doc_{version}-1_all.deb
+ # - libarrow-glib0-dbgsym_{version}-1_amd64.deb
+ # - libarrow-glib0_{version}-1_amd64.deb
+ # - libarrow-gpu-dev_{version}-1_amd64.deb
+ # - libarrow-gpu-glib-dev_{version}-1_amd64.deb
+ # - libarrow-gpu-glib0-dbgsym_{version}-1_amd64.deb
+ # - libarrow-gpu-glib0_{version}-1_amd64.deb
+ # - libarrow-gpu0-dbgsym_{version}-1_amd64.deb
+ # - libarrow-gpu0_{version}-1_amd64.deb
+ # - libarrow-python-dev_{version}-1_amd64.deb
+ # - libarrow-python0-dbgsym_{version}-1_amd64.deb
+ # - libarrow-python0_{version}-1_amd64.deb
+ # - libarrow0-dbgsym_{version}-1_amd64.deb
+ # - libarrow0_{version}-1_amd64.deb
+
+ # YUM
+ - arrow-{version}-1.el6.src.rpm
+ - arrow-{version}-1.el7.src.rpm
+ - arrow-debuginfo-{version}-1.el6.x86_64.rpm
+ - arrow-debuginfo-{version}-1.el7.x86_64.rpm
+ - arrow-devel-{version}-1.el6.x86_64.rpm
+ - arrow-devel-{version}-1.el7.x86_64.rpm
+ - arrow-glib-devel-{version}-1.el7.x86_64.rpm
+ - arrow-glib-doc-{version}-1.el7.x86_64.rpm
+ - arrow-glib-libs-{version}-1.el7.x86_64.rpm
+ - arrow-libs-{version}-1.el6.x86_64.rpm
+ - arrow-libs-{version}-1.el7.x86_64.rpm
+ - arrow-python-devel-{version}-1.el6.x86_64.rpm
+ - arrow-python-devel-{version}-1.el7.x86_64.rpm
+ - arrow-python-libs-{version}-1.el6.x86_64.rpm
+ - arrow-python-libs-{version}-1.el7.x86_64.rpm