You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by ap...@apache.org on 2019/06/26 11:51:39 UTC
[arrow] branch master updated: ARROW-5725: [Crossbow] Port conda
recipes to azure pipelines
This is an automated email from the ASF dual-hosted git repository.
apitrou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/master by this push:
new ebff606 ARROW-5725: [Crossbow] Port conda recipes to azure pipelines
ebff606 is described below
commit ebff606ec47308e5ef260a45d6c2ea14bb745718
Author: Krisztián Szűcs <sz...@gmail.com>
AuthorDate: Wed Jun 26 13:51:30 2019 +0200
ARROW-5725: [Crossbow] Port conda recipes to azure pipelines
- [x] artifact uploading
- [x] osx build
- [x] win build (using appveyor, because of https://github.com/conda-forge/conda-forge.github.io/issues/703)
- [x] linux build
- [x] package gandiva
Author: Krisztián Szűcs <sz...@gmail.com>
Closes #4649 from kszucs/crossbow-azure and squashes the following commits:
20f6cecd3 <Krisztián Szűcs> update conda-win artifact patterns
0c470bb6a <Krisztián Szűcs> readme fixes
b6a86076b <Krisztián Szűcs> configure output folder for the artifacts
68d88a833 <Krisztián Szűcs> combine status api and checks api
111964957 <Krisztián Szűcs> fix artifact patterns
95cb44217 <Krisztián Szűcs> use FETCH_HEAD in the CI templates
cbb9c9ce7 <Krisztián Szűcs> rat
8f58839e1 <Krisztián Szűcs> use the default python on osx
f75efae18 <Krisztián Szűcs> use pip module for installing dependencies
2a598945d <Krisztián Szűcs> tabulate win template
9db3db1dd <Krisztián Szűcs> use pip3
2aa497748 <Krisztián Szűcs> azure template for docker tests
750f624c1 <Krisztián Szűcs> asset uploading script
e0d8fb9b2 <Krisztián Szűcs> git commit additional log
7fbce5df8 <Krisztián Szűcs> use appveyor for the win packages; upload assets scripts
d6c4ce9fa <Krisztián Szűcs> touch done_canary
611222e28 <Krisztián Szűcs> docker shm
ba0e88cce <Krisztián Szűcs> update old templates; query cxx include paths
0d76f1364 <Krisztián Szűcs> win
0c8464a4b <Krisztián Szűcs> parquet-cpp depend on exact arrow-cpp version
aecc2b19e <Krisztián Szűcs> displayName order
c42ebf595 <Krisztián Szűcs> quoting gandiva flags
8abd34779 <Krisztián Szűcs> move displayName after the script
bdf705ff0 <Krisztián Szűcs> OSX configuration
a874d1f99 <Krisztián Szűcs> gandiva flags
f50af1b51 <Krisztián Szűcs> path gymnastics
5cd9fa0b5 <Krisztián Szűcs> use pyarrow as recipe root
4b005892f <Krisztián Szűcs> try to fix assertion error
9ef81c567 <Krisztián Szűcs> use feedstock_root
0e826ac43 <Krisztián Szűcs> fix recipe directories
adae7c0f3 <Krisztián Szűcs> build all three recipes
7b60c9d07 <Krisztián Szűcs> pass arrow_version
ce740d799 <Krisztián Szűcs> fixing build_steps.sh path
df31ff7dc <Krisztián Szűcs> trying to fix feedstock and recipe roots
501d55341 <Krisztián Szűcs> set config
b2425e650 <Krisztián Szűcs> fix working directory
53e8eb24f <Krisztián Szűcs> don't use azure templates
b2fd21a24 <Krisztián Szűcs> use variables ]
2037f78fc <Krisztián Szűcs> port conda recipes to azure pipelines
---
LICENSE.txt | 2 +-
dev/release/rat_exclude_files.txt | 2 +-
.../{variants => .ci_support}/linux_python2.7.yaml | 26 +----
.../{variants => .ci_support}/linux_python3.6.yaml | 26 +----
.../{variants => .ci_support}/linux_python3.7.yaml | 27 ++----
.../{variants => .ci_support}/osx_python2.7.yaml | 26 +----
.../{variants => .ci_support}/osx_python3.6.yaml | 26 +----
.../{variants => .ci_support}/osx_python3.7.yaml | 26 +----
...compilervs2015cxx_compilervs2015python3.6.yaml} | 27 +-----
...compilervs2015cxx_compilervs2015python3.7.yaml} | 27 +-----
dev/tasks/conda-recipes/README.md | 69 +++++++++++++
.../{appveyor.yml => appveyor.win.yml} | 21 +---
dev/tasks/conda-recipes/arrow-cpp/bld.bat | 17 ----
dev/tasks/conda-recipes/arrow-cpp/build.sh | 29 ++----
dev/tasks/conda-recipes/arrow-cpp/meta.yaml | 20 +---
dev/tasks/conda-recipes/azure.linux.yml | 51 ++++++++++
dev/tasks/conda-recipes/azure.osx.yml | 84 ++++++++++++++++
dev/tasks/conda-recipes/azure.win.yml | 108 +++++++++++++++++++++
dev/tasks/conda-recipes/build_steps.sh | 39 ++++++++
dev/tasks/conda-recipes/parquet-cpp/meta.yaml | 22 +----
dev/tasks/conda-recipes/pyarrow/bld.bat | 17 ----
dev/tasks/conda-recipes/pyarrow/build.sh | 17 ----
dev/tasks/conda-recipes/pyarrow/meta.yaml | 17 ----
dev/tasks/conda-recipes/run_docker_build.sh | 71 ++++++++++++++
dev/tasks/conda-recipes/travis.linux.yml | 78 ---------------
dev/tasks/conda-recipes/travis.osx.yml | 83 ----------------
dev/tasks/crossbow.py | 102 ++++++++++++++++---
.../build.sh => docker-tests/azure.linux.yml} | 42 ++++----
dev/tasks/tasks.yml | 35 +++----
dev/tasks/upload-assets.py | 64 ++++++++++++
30 files changed, 666 insertions(+), 535 deletions(-)
diff --git a/LICENSE.txt b/LICENSE.txt
index 1be16cc..df5f5cc 100644
--- a/LICENSE.txt
+++ b/LICENSE.txt
@@ -740,7 +740,7 @@ You can contact the author at :
--------------------------------------------------------------------------------
-The files in dev/tasks/conda-recipes/variants have the following license
+The files under dev/tasks/conda-recipes have the following license
BSD 3-clause license
Copyright (c) 2015-2018, conda-forge
diff --git a/dev/release/rat_exclude_files.txt b/dev/release/rat_exclude_files.txt
index 25b1ecd..1d7973d 100644
--- a/dev/release/rat_exclude_files.txt
+++ b/dev/release/rat_exclude_files.txt
@@ -130,7 +130,7 @@ dev/tasks/linux-packages/debian/plasma-store-server.install
dev/tasks/linux-packages/debian/rules
dev/tasks/linux-packages/debian/source/format
dev/tasks/linux-packages/debian/watch
-dev/tasks/conda-recipes/variants/*.yaml
+dev/tasks/conda-recipes/*
docs/requirements.txt
go/arrow/go.sum
go/arrow/Gopkg.lock
diff --git a/dev/tasks/conda-recipes/variants/linux_python2.7.yaml b/dev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
similarity index 54%
rename from dev/tasks/conda-recipes/variants/linux_python2.7.yaml
rename to dev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
index d92357b..8757d94 100644
--- a/dev/tasks/conda-recipes/variants/linux_python2.7.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/linux_python2.7.yaml
@@ -1,40 +1,24 @@
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- gcc
+c_compiler_version:
+- '7'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- gxx
+cxx_compiler_version:
+- '7'
docker_image:
- condaforge/linux-anvil-comp7
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '2.7'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/linux_python3.6.yaml b/dev/tasks/conda-recipes/.ci_support/linux_python3.6.yaml
similarity index 54%
rename from dev/tasks/conda-recipes/variants/linux_python3.6.yaml
rename to dev/tasks/conda-recipes/.ci_support/linux_python3.6.yaml
index a934bd1..6af2d07 100644
--- a/dev/tasks/conda-recipes/variants/linux_python3.6.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/linux_python3.6.yaml
@@ -1,40 +1,24 @@
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- gcc
+c_compiler_version:
+- '7'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- gxx
+cxx_compiler_version:
+- '7'
docker_image:
- condaforge/linux-anvil-comp7
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.6'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/linux_python3.7.yaml b/dev/tasks/conda-recipes/.ci_support/linux_python3.7.yaml
similarity index 54%
rename from dev/tasks/conda-recipes/variants/linux_python3.7.yaml
rename to dev/tasks/conda-recipes/.ci_support/linux_python3.7.yaml
index 736cd01..1a27da2 100644
--- a/dev/tasks/conda-recipes/variants/linux_python3.7.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/linux_python3.7.yaml
@@ -1,40 +1,25 @@
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- gcc
+c_compiler_version:
+- '7'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- gxx
+cxx_compiler_version:
+- '7'
docker_image:
- condaforge/linux-anvil-comp7
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.7'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
+
diff --git a/dev/tasks/conda-recipes/variants/osx_python2.7.yaml b/dev/tasks/conda-recipes/.ci_support/osx_python2.7.yaml
similarity index 60%
rename from dev/tasks/conda-recipes/variants/osx_python2.7.yaml
rename to dev/tasks/conda-recipes/.ci_support/osx_python2.7.yaml
index 2817157..771f49f 100644
--- a/dev/tasks/conda-recipes/variants/osx_python2.7.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/osx_python2.7.yaml
@@ -1,19 +1,19 @@
MACOSX_DEPLOYMENT_TARGET:
- '10.9'
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- clang
+c_compiler_version:
+- '4'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- clangxx
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
+cxx_compiler_version:
+- '4'
macos_machine:
- x86_64-apple-darwin13.4.0
macos_min_version:
@@ -21,24 +21,8 @@ macos_min_version:
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '2.7'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/osx_python3.6.yaml b/dev/tasks/conda-recipes/.ci_support/osx_python3.6.yaml
similarity index 60%
rename from dev/tasks/conda-recipes/variants/osx_python3.6.yaml
rename to dev/tasks/conda-recipes/.ci_support/osx_python3.6.yaml
index 5e87a2d..ed2e5a0 100644
--- a/dev/tasks/conda-recipes/variants/osx_python3.6.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/osx_python3.6.yaml
@@ -1,19 +1,19 @@
MACOSX_DEPLOYMENT_TARGET:
- '10.9'
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- clang
+c_compiler_version:
+- '4'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- clangxx
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
+cxx_compiler_version:
+- '4'
macos_machine:
- x86_64-apple-darwin13.4.0
macos_min_version:
@@ -21,24 +21,8 @@ macos_min_version:
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.6'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/osx_python3.7.yaml b/dev/tasks/conda-recipes/.ci_support/osx_python3.7.yaml
similarity index 60%
rename from dev/tasks/conda-recipes/variants/osx_python3.7.yaml
rename to dev/tasks/conda-recipes/.ci_support/osx_python3.7.yaml
index 631716d..5a0946c 100644
--- a/dev/tasks/conda-recipes/variants/osx_python3.7.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/osx_python3.7.yaml
@@ -1,19 +1,19 @@
MACOSX_DEPLOYMENT_TARGET:
- '10.9'
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- clang
+c_compiler_version:
+- '4'
channel_sources:
- conda-forge,defaults
channel_targets:
- conda-forge main
cxx_compiler:
- clangxx
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
+cxx_compiler_version:
+- '4'
macos_machine:
- x86_64-apple-darwin13.4.0
macos_min_version:
@@ -21,24 +21,8 @@ macos_min_version:
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.7'
-snappy:
-- 1.1.7
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6vc14.yaml b/dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
similarity index 52%
rename from dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6vc14.yaml
rename to dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
index b87a06e..9961c53 100644
--- a/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.6vc14.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.6.yaml
@@ -1,5 +1,5 @@
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- vs2015
channel_sources:
@@ -8,40 +8,15 @@ channel_targets:
- conda-forge main
cxx_compiler:
- vs2015
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- vc:
- max_pin: x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.6'
-snappy:
-- 1.1.7
-vc:
-- '14'
zip_keys:
- - python
- - vc
- c_compiler
- cxx_compiler
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.7vc14.yaml b/dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.7.yaml
similarity index 52%
rename from dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.7vc14.yaml
rename to dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.7.yaml
index 5c710ff..77abc37 100644
--- a/dev/tasks/conda-recipes/variants/win_c_compilervs2015cxx_compilervs2015python3.7vc14.yaml
+++ b/dev/tasks/conda-recipes/.ci_support/win_c_compilervs2015cxx_compilervs2015python3.7.yaml
@@ -1,5 +1,5 @@
boost_cpp:
-- 1.68.0
+- 1.70.0
c_compiler:
- vs2015
channel_sources:
@@ -8,40 +8,15 @@ channel_targets:
- conda-forge main
cxx_compiler:
- vs2015
-libprotobuf:
-- '3.7'
-lz4_c:
-- 1.8.1
pin_run_as_build:
boost-cpp:
max_pin: x.x.x
- libprotobuf:
- max_pin: x.x
- lz4-c:
- max_pin: x.x.x
python:
min_pin: x.x
max_pin: x.x
- snappy:
- max_pin: x.x.x
- vc:
- max_pin: x
- zlib:
- max_pin: x.x
- zstd:
- max_pin: x.x.x
python:
- '3.7'
-snappy:
-- 1.1.7
-vc:
-- '14'
zip_keys:
- - python
- - vc
- c_compiler
- cxx_compiler
-zlib:
-- '1.2'
-zstd:
-- 1.3.3
diff --git a/dev/tasks/conda-recipes/README.md b/dev/tasks/conda-recipes/README.md
new file mode 100644
index 0000000..d5cfe3e
--- /dev/null
+++ b/dev/tasks/conda-recipes/README.md
@@ -0,0 +1,69 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Conda Forge recipes
+
+This directory must be migrated periodically with the upstrem updates of
+[arrow-cpp-feedstock][arrow-cpp-feedsotkc],
+[parquet-cpp-feedstock][parquet-cpp-feedstock] and
+[pyarrow-feedstock][pyarrow-feedstock]
+conda-forge repositories because of multiple vendored files.
+
+## Keeping the recipes synchronized
+
+The recipes here are tested on nightly basis, so they follow the development
+versions of arrow instead of the upstream recipes, which are suitable for the
+latest releases.
+
+### Backporting from the upstream feedstocks
+
+In most of the cases these recipes are more accurate, then the upstream
+feedstocks. Altough the upstream feedstocks regurarly receive automatic updates
+by the conda-forge team so we need to backport those changes to the crossbow
+recipes. Most of these updates are touching the version pinning files
+(under `.ci_support`) and other CI related configuration files.
+
+Because all three recipes must be built in the same continuous integration
+job prefer porting from the [pyarrow feedstock][pyarrow-feedstock].
+
+#### Updating the variants:
+
+Copy the configuration files from `pyarrow-feedstock/.ci_support` to the
+`.ci_support` folder.
+
+#### Updating the CI configurations:
+
+The `.azure-pipelines/azure-pipelines-[linux|osx|win].yml` should be ported
+to the local counterparts under `.azure-pipelines` with keeping the crossbow
+related parts (the cloning of arrow and the jinja templated variables) and
+moving the matrix definitions like [this][matrix-definition] to the crossbow
+[tasks.yml][../tasks.yml] config file.
+
+
+### Porting recipes from crossbow to the upstream feedstocks
+
+Theoretically these recipes should be up to date with the actual version of
+Arrow, so during the release procedure the content of these recipes should be
+copied to the upstream feedstocks.
+
+
+[arrow-cpp-feedstock]: https://github.com/conda-forge/arrow-cpp-feedstock
+[parquet-cpp-feedstock]: https://github.com/conda-forge/parquet-cpp-feedstock
+[pyarrow-cpp-feedstock]: https://github.com/conda-forge/pyarrow-feedstock
+[matrix-definition]: https://github.com/conda-forge/pyarrow-feedstock/blob/master/.azure-pipelines/azure-pipelines-linux.yml#L12
diff --git a/dev/tasks/conda-recipes/appveyor.yml b/dev/tasks/conda-recipes/appveyor.win.yml
similarity index 63%
rename from dev/tasks/conda-recipes/appveyor.yml
rename to dev/tasks/conda-recipes/appveyor.win.yml
index 5d677c6..121595a 100644
--- a/dev/tasks/conda-recipes/appveyor.yml
+++ b/dev/tasks/conda-recipes/appveyor.win.yml
@@ -1,20 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
environment:
ARROW_VERSION: {{ arrow.no_rc_version }}
# regardless of the python version we build against
@@ -50,11 +33,11 @@ test_script:
- pushd arrow\dev\tasks\conda-recipes
# Configure conda
- - cmd: setup_conda_rc .\ .\ variants\{{ config }}.yaml
+ - cmd: setup_conda_rc .\ .\ .ci_support\{{ config }}.yaml
- cmd: run_conda_forge_build_setup
# Build the recipes
- - conda.exe build --output-folder . -m variants\{{ config }}.yaml parquet-cpp arrow-cpp pyarrow
+ - conda.exe build --output-folder . -m .ci_support\{{ config }}.yaml parquet-cpp arrow-cpp pyarrow
# Rename artifacts
- pushd win-64
diff --git a/dev/tasks/conda-recipes/arrow-cpp/bld.bat b/dev/tasks/conda-recipes/arrow-cpp/bld.bat
index c853a63..429cb64 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/bld.bat
+++ b/dev/tasks/conda-recipes/arrow-cpp/bld.bat
@@ -1,20 +1,3 @@
-@rem Licensed to the Apache Software Foundation (ASF) under one
-@rem or more contributor license agreements. See the NOTICE file
-@rem distributed with this work for additional information
-@rem regarding copyright ownership. The ASF licenses this file
-@rem to you under the Apache License, Version 2.0 (the
-@rem "License"); you may not use this file except in compliance
-@rem with the License. You may obtain a copy of the License at
-@rem
-@rem http://www.apache.org/licenses/LICENSE-2.0
-@rem
-@rem Unless required by applicable law or agreed to in writing,
-@rem software distributed under the License is distributed on an
-@rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-@rem KIND, either express or implied. See the License for the
-@rem specific language governing permissions and limitations
-@rem under the License.
-
mkdir "%SRC_DIR%"\cpp\build
pushd "%SRC_DIR%"\cpp\build
diff --git a/dev/tasks/conda-recipes/arrow-cpp/build.sh b/dev/tasks/conda-recipes/arrow-cpp/build.sh
index dfe992e..b18862a 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/build.sh
+++ b/dev/tasks/conda-recipes/arrow-cpp/build.sh
@@ -1,22 +1,3 @@
-#!/bin/sh
-
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
set -e
set -x
@@ -24,6 +5,14 @@ cd cpp
mkdir build-dir
cd build-dir
+EXTRA_CMAKE_ARGS=""
+
+# Include g++'s system headers
+if [ "$(uname)" == "Linux" ]; then
+ SYSTEM_INCLUDES=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
+ EXTRA_CMAKE_ARGS=" -DARROW_GANDIVA_PC_CXX_FLAGS=${SYSTEM_INCLUDES}"
+fi
+
cmake \
-DCMAKE_BUILD_TYPE=release \
-DCMAKE_INSTALL_PREFIX=$PREFIX \
@@ -41,10 +30,10 @@ cmake \
-DARROW_PARQUET=ON \
-DARROW_GANDIVA=ON \
-DARROW_ORC=ON \
- -DORC_HOME=$PREFIX \
-DCMAKE_AR=${AR} \
-DCMAKE_RANLIB=${RANLIB} \
-GNinja \
+ ${EXTRA_CMAKE_ARGS} \
..
ninja install
diff --git a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
index 15fc9ea..877bfc6 100644
--- a/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
+++ b/dev/tasks/conda-recipes/arrow-cpp/meta.yaml
@@ -1,20 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
package:
name: arrow-cpp
version: {{ ARROW_VERSION }}
@@ -31,8 +14,8 @@ build:
requirements:
build:
- - autoconf # [unix]
- cmake
+ - autoconf # [unix]
- ninja
- {{ compiler('c') }}
- {{ compiler('cxx') }}
@@ -61,6 +44,7 @@ requirements:
- {{ pin_compatible('numpy', lower_bound='1.14') }}
- boost-cpp
- brotli
+ - double-conversion
- gflags
- glog
- lz4-c
diff --git a/dev/tasks/conda-recipes/azure.linux.yml b/dev/tasks/conda-recipes/azure.linux.yml
new file mode 100755
index 0000000..355f3b1
--- /dev/null
+++ b/dev/tasks/conda-recipes/azure.linux.yml
@@ -0,0 +1,51 @@
+jobs:
+- job: linux
+ pool:
+ vmImage: ubuntu-16.04
+ timeoutInMinutes: 360
+
+ variables:
+ CONFIG: {{ config }}
+ ARROW_VERSION: {{ arrow.no_rc_version }}
+ DOCKER_IMAGE: condaforge/linux-anvil-comp7
+ UPLOAD_PACKAGES: False
+
+ steps:
+ # configure qemu binfmt-misc running. This allows us to run docker containers
+ # embedded qemu-static
+ - script: |
+ docker run --rm --privileged multiarch/qemu-user-static:register --reset --credential yes
+ ls /proc/sys/fs/binfmt_misc/
+ displayName: Configure binfmt_misc
+ condition: not(startsWith(variables['CONFIG'], 'linux_64'))
+
+ - script: |
+ git clone --no-checkout {{ arrow.remote }} arrow
+ git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ git -C arrow checkout FETCH_HEAD
+ displayName: Clone arrow
+
+ - script: CI=azure ./run_docker_build.sh
+ displayName: Run docker build
+ workingDirectory: arrow/dev/tasks/conda-recipes
+
+ # Using github release tries to find a common ancestor between the currently
+ # pushed tag and the latest tag of the github repository (don't know why).
+ # The tag upload took 43 minutes because of this scan, so use an alternative
+ # upload script.
+ - task: UsePythonVersion@0
+ inputs:
+ versionSpec: '3.6'
+ architecture: 'x64'
+
+ - script: |
+ python -m pip install github3-py click
+ python upload-assets.py \
+ --sha {{ task.branch }} \
+ --tag {{ task.tag }} \
+ --pattern "conda-recipes/build_artifacts/linux-64/*.tar.bz2"
+ env:
+ CROSSBOW_GITHUB_REPO: $(Build.Repository.Name)
+ CROSSBOW_GITHUB_TOKEN: $(CROSSBOW_GITHUB_TOKEN)
+ displayName: Upload packages as a GitHub release
+ workingDirectory: arrow/dev/tasks
diff --git a/dev/tasks/conda-recipes/azure.osx.yml b/dev/tasks/conda-recipes/azure.osx.yml
new file mode 100755
index 0000000..5f5c8d0
--- /dev/null
+++ b/dev/tasks/conda-recipes/azure.osx.yml
@@ -0,0 +1,84 @@
+jobs:
+- job: osx
+ pool:
+ vmImage: macOS-10.13
+ timeoutInMinutes: 360
+ variables:
+ CONFIG: {{ config }}
+ ARROW_VERSION: {{ arrow.no_rc_version }}
+ UPLOAD_PACKAGES: False
+ steps:
+ - script: |
+ echo "Removing homebrew from Azure to avoid conflicts."
+ curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall > ~/uninstall_homebrew
+ chmod +x ~/uninstall_homebrew
+ ~/uninstall_homebrew -fq
+ rm ~/uninstall_homebrew
+ displayName: Remove homebrew
+
+ - bash: |
+ echo "##vso[task.prependpath]$CONDA/bin"
+ sudo chown -R $USER $CONDA
+ displayName: Add conda to PATH
+
+ - script: |
+ source activate base
+ conda install -n base -c conda-forge --quiet --yes conda-forge-ci-setup=2 conda-build
+ displayName: 'Add conda-forge-ci-setup=2'
+
+ - script: |
+ git clone --no-checkout {{ arrow.remote }} arrow
+ git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ git -C arrow checkout FETCH_HEAD
+ displayName: Clone arrow
+
+ - script: |
+ source activate base
+ echo "Configuring conda."
+
+ setup_conda_rc ./ ./ ./.ci_support/${CONFIG}.yaml
+ export CI=azure
+ source run_conda_forge_build_setup
+ conda update --yes --quiet --override-channels -c conda-forge -c defaults --all
+ displayName: Configure conda and conda-build
+ workingDirectory: arrow/dev/tasks/conda-recipes
+ env: {
+ OSX_FORCE_SDK_DOWNLOAD: "1"
+ }
+
+ - script: |
+ source activate base
+ mangle_compiler ./ ./ ./.ci_support/${CONFIG}.yaml
+ workingDirectory: arrow/dev/tasks/conda-recipes
+ displayName: Mangle compiler
+
+ - script: |
+ source activate base
+ make_build_number ./ ./ ./.ci_support/${CONFIG}.yaml
+ workingDirectory: arrow/dev/tasks/conda-recipes
+ displayName: Generate build number clobber file
+
+ - script: |
+ source activate base
+ conda build arrow-cpp parquet-cpp pyarrow \
+ -m ./.ci_support/${CONFIG}.yaml \
+ --clobber-file ./.ci_support/clobber_${CONFIG}.yaml \
+ --output-folder ./build_artifacts
+ workingDirectory: arrow/dev/tasks/conda-recipes
+ displayName: Build recipes
+
+ # Using github release tries to find a common ancestor between the currently
+ # pushed tag and the latest tag of the github repository (don't know why).
+ # The tag upload took 43 minutes because of this scan, so use an alternative
+ # upload script.
+ - script: |
+ python -m pip install github3-py click
+ python upload-assets.py \
+ --sha {{ task.branch }} \
+ --tag {{ task.tag }} \
+ --pattern "conda-recipes/build_artifacts/osx-64/*.tar.bz2"
+ env:
+ CROSSBOW_GITHUB_REPO: $(Build.Repository.Name)
+ CROSSBOW_GITHUB_TOKEN: $(CROSSBOW_GITHUB_TOKEN)
+ displayName: Upload packages as a GitHub release
+ workingDirectory: arrow/dev/tasks
diff --git a/dev/tasks/conda-recipes/azure.win.yml b/dev/tasks/conda-recipes/azure.win.yml
new file mode 100755
index 0000000..02fc91a
--- /dev/null
+++ b/dev/tasks/conda-recipes/azure.win.yml
@@ -0,0 +1,108 @@
+jobs:
+- job: win
+ pool:
+ vmImage: vs2017-win2016
+ timeoutInMinutes: 360
+ variables:
+ CONFIG: {{ config }}
+ ARROW_VERSION: {{ arrow.no_rc_version }}
+ CONDA_BLD_PATH: D:\\bld\\
+ UPLOAD_PACKAGES: False
+ steps:
+ - script: |
+ choco install vcpython27 -fdv -y --debug
+ condition: contains(variables['CONFIG'], 'vs2008')
+ displayName: Install vcpython27.msi (if needed)
+
+ - powershell: |
+ Set-PSDebug -Trace 1
+
+ $batchcontent = @"
+ ECHO ON
+ SET vcpython=C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0
+
+ DIR "%vcpython%"
+
+ CALL "%vcpython%\vcvarsall.bat" %*
+ "@
+
+ $batchDir = "C:\Program Files (x86)\Common Files\Microsoft\Visual C++ for Python\9.0\VC"
+ $batchPath = "$batchDir" + "\vcvarsall.bat"
+ New-Item -Path $batchPath -ItemType "file" -Force
+
+ Set-Content -Value $batchcontent -Path $batchPath
+
+ Get-ChildItem -Path $batchDir
+
+ Get-ChildItem -Path ($batchDir + '\..')
+
+ condition: contains(variables['CONFIG'], 'vs2008')
+ displayName: Patch vs2008 (if needed)
+
+ - task: CondaEnvironment@1
+ inputs:
+ packageSpecs: 'python=3.6 conda-build conda conda-forge::conda-forge-ci-setup=2' # Optional
+ installOptions: "-c conda-forge"
+ updateConda: false
+ displayName: Install conda-build and activate environment
+
+ - script: set PYTHONUNBUFFERED=1
+
+ - script: |
+ git clone --no-checkout {{ arrow.remote }} arrow
+ git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ git -C arrow checkout FETCH_HEAD
+ displayName: Clone arrow
+
+ # Configure the VM
+ - script: setup_conda_rc .\ .\ .\.ci_support\%CONFIG%.yaml
+ workingDirectory: arrow\dev\tasks\conda-recipes
+
+ # Configure the VM.
+ - script: |
+ set "CI=azure"
+ run_conda_forge_build_setup
+ displayName: conda-forge build setup
+
+ - script: |
+ rmdir C:\strawberry /s /q
+ continueOnError: true
+ displayName: remove strawberryperl
+
+ # Special cased version setting some more things!
+ - script: |
+ conda.exe build arrow-cpp parquet-cpp pyarrow -m .ci_support\%CONFIG%.yaml
+ displayName: Build recipe (vs2008)
+ workingDirectory: arrow\dev\tasks\conda-recipes
+ env:
+ VS90COMNTOOLS: "C:\\Program Files (x86)\\Common Files\\Microsoft\\Visual C++ for Python\\9.0\\VC\\bin"
+ PYTHONUNBUFFERED: 1
+ condition: contains(variables['CONFIG'], 'vs2008')
+
+ - script: |
+ conda.exe build arrow-cpp parquet-cpp pyarrow -m .ci_support\%CONFIG%.yaml
+ displayName: Build recipe
+ workingDirectory: arrow\dev\tasks\conda-recipes
+ env:
+ PYTHONUNBUFFERED: 1
+ condition: not(contains(variables['CONFIG'], 'vs2008'))
+
+ # Using github release tries to find a common ancestor between the currently
+ # pushed tag and the latest tag of the github repository (don't know why).
+ # The tag upload took 43 minutes because of this scan, so use an alternative upload script.
+ - task: UsePythonVersion@0
+ inputs:
+ versionSpec: '3.6'
+ architecture: 'x64'
+
+ - script: |
+ python -m pip install github3-py click
+ python upload-assets.py \
+ --sha {{ task.branch }} \
+ --tag {{ task.tag }} \
+ --pattern "conda-recipes/build_artifacts/*.tar.bz2"
+ env:
+ CROSSBOW_GITHUB_REPO: $(Build.Repository.Name)
+ CROSSBOW_GITHUB_TOKEN: $(CROSSBOW_GITHUB_TOKEN)
+ displayName: Upload packages as a GitHub release
+ workingDirectory: arrow/dev/tasks
diff --git a/dev/tasks/conda-recipes/build_steps.sh b/dev/tasks/conda-recipes/build_steps.sh
new file mode 100755
index 0000000..9f243f3
--- /dev/null
+++ b/dev/tasks/conda-recipes/build_steps.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+# PLEASE NOTE: This script has been automatically generated by conda-smithy. Any changes here
+# will be lost next time ``conda smithy rerender`` is run. If you would like to make permanent
+# changes to this script, consider a proposal to conda-smithy so that other feedstocks can also
+# benefit from the improvement.
+
+set -xeuo pipefail
+export PYTHONUNBUFFERED=1
+export FEEDSTOCK_ROOT="${FEEDSTOCK_ROOT:-/home/conda/feedstock_root}"
+export CI_SUPPORT="${FEEDSTOCK_ROOT}/.ci_support"
+export CONFIG_FILE="${CI_SUPPORT}/${CONFIG}.yaml"
+
+cat >~/.condarc <<CONDARC
+
+conda-build:
+ root-dir: ${FEEDSTOCK_ROOT}/build_artifacts
+
+CONDARC
+
+conda install --yes --quiet conda-forge-ci-setup=2 conda-build -c conda-forge
+
+# set up the condarc
+setup_conda_rc "${FEEDSTOCK_ROOT}" "${FEEDSTOCK_ROOT}" "${CONFIG_FILE}"
+
+source run_conda_forge_build_setup
+
+# make the build number clobber
+make_build_number "${FEEDSTOCK_ROOT}" "${FEEDSTOCK_ROOT}" "${CONFIG_FILE}"
+
+conda build \
+ "${FEEDSTOCK_ROOT}/arrow-cpp" \
+ "${FEEDSTOCK_ROOT}/parquet-cpp" \
+ "${FEEDSTOCK_ROOT}/pyarrow" \
+ -m "${CI_SUPPORT}/${CONFIG}.yaml" \
+ --clobber-file "${CI_SUPPORT}/clobber_${CONFIG}.yaml" \
+ --output-folder "${FEEDSTOCK_ROOT}/build_artifacts"
+
+touch "${FEEDSTOCK_ROOT}/build_artifacts/conda-forge-build-done-${CONFIG}"
diff --git a/dev/tasks/conda-recipes/parquet-cpp/meta.yaml b/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
index eebedd8..9bf0c9c 100644
--- a/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
+++ b/dev/tasks/conda-recipes/parquet-cpp/meta.yaml
@@ -1,20 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
# ARROW-3229: this is a meta-package to prevent conflicts in the future
{% set parquet_version = "1.5.1" %}
@@ -30,9 +13,10 @@ build:
requirements:
host:
- - arrow-cpp >={{ ARROW_VERSION }}
+ # NOTE: in the upstream feedstock use >= instead of =
+ - arrow-cpp ={{ ARROW_VERSION }}
run:
- - arrow-cpp >={{ ARROW_VERSION }}
+ - arrow-cpp ={{ ARROW_VERSION }}
test:
commands:
diff --git a/dev/tasks/conda-recipes/pyarrow/bld.bat b/dev/tasks/conda-recipes/pyarrow/bld.bat
index 37d7f18..d02ceaf 100644
--- a/dev/tasks/conda-recipes/pyarrow/bld.bat
+++ b/dev/tasks/conda-recipes/pyarrow/bld.bat
@@ -1,20 +1,3 @@
-@rem Licensed to the Apache Software Foundation (ASF) under one
-@rem or more contributor license agreements. See the NOTICE file
-@rem distributed with this work for additional information
-@rem regarding copyright ownership. The ASF licenses this file
-@rem to you under the Apache License, Version 2.0 (the
-@rem "License"); you may not use this file except in compliance
-@rem with the License. You may obtain a copy of the License at
-@rem
-@rem http://www.apache.org/licenses/LICENSE-2.0
-@rem
-@rem Unless required by applicable law or agreed to in writing,
-@rem software distributed under the License is distributed on an
-@rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-@rem KIND, either express or implied. See the License for the
-@rem specific language governing permissions and limitations
-@rem under the License.
-
pushd "%SRC_DIR%"\python
@rem the symlinks for cmake modules don't work here
diff --git a/dev/tasks/conda-recipes/pyarrow/build.sh b/dev/tasks/conda-recipes/pyarrow/build.sh
index 56d6fa7..98b10a4 100644
--- a/dev/tasks/conda-recipes/pyarrow/build.sh
+++ b/dev/tasks/conda-recipes/pyarrow/build.sh
@@ -1,22 +1,5 @@
#!/bin/sh
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
set -e
set -x
diff --git a/dev/tasks/conda-recipes/pyarrow/meta.yaml b/dev/tasks/conda-recipes/pyarrow/meta.yaml
index 4c8f656..8198390 100644
--- a/dev/tasks/conda-recipes/pyarrow/meta.yaml
+++ b/dev/tasks/conda-recipes/pyarrow/meta.yaml
@@ -1,20 +1,3 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
package:
name: pyarrow
version: {{ ARROW_VERSION }}
diff --git a/dev/tasks/conda-recipes/run_docker_build.sh b/dev/tasks/conda-recipes/run_docker_build.sh
new file mode 100755
index 0000000..99476c5
--- /dev/null
+++ b/dev/tasks/conda-recipes/run_docker_build.sh
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+# PLEASE NOTE: This script has been automatically generated by conda-smithy. Any changes here
+# will be lost next time ``conda smithy rerender`` is run. If you would like to make permanent
+# changes to this script, consider a proposal to conda-smithy so that other feedstocks can also
+# benefit from the improvement.
+
+set -xeo pipefail
+
+THISDIR="$( cd "$( dirname "$0" )" >/dev/null && pwd )"
+ARROW_ROOT=$(cd "$THISDIR/../../.."; pwd;)
+FEEDSTOCK_ROOT=$THISDIR
+
+docker info
+
+# In order for the conda-build process in the container to write to the mounted
+# volumes, we need to run with the same id as the host machine, which is
+# normally the owner of the mounted volumes, or at least has write permission
+export HOST_USER_ID=$(id -u)
+# Check if docker-machine is being used (normally on OSX) and get the uid from
+# the VM
+if hash docker-machine 2> /dev/null && docker-machine active > /dev/null; then
+ export HOST_USER_ID=$(docker-machine ssh $(docker-machine active) id -u)
+fi
+
+ARTIFACTS="$FEEDSTOCK_ROOT/build_artifacts"
+
+if [ -z "$CONFIG" ]; then
+ set +x
+ FILES=`ls .ci_support/linux_*`
+ CONFIGS=""
+ for file in $FILES; do
+ CONFIGS="${CONFIGS}'${file:12:-5}' or ";
+ done
+ echo "Need to set CONFIG env variable. Value can be one of ${CONFIGS:0:-4}"
+ exit 1
+fi
+
+if [ -z "${DOCKER_IMAGE}" ]; then
+ SHYAML_INSTALLED="$(shyaml --version || echo NO)"
+ if [ "${SHYAML_INSTALLED}" == "NO" ]; then
+ echo "WARNING: DOCKER_IMAGE variable not set and shyaml not installed. Falling back to condaforge/linux-anvil-comp7"
+ DOCKER_IMAGE="condaforge/linux-anvil-comp7"
+ else
+ DOCKER_IMAGE="$(cat "${FEEDSTOCK_ROOT}/.ci_support/${CONFIG}.yaml" | shyaml get-value docker_image.0 condaforge/linux-anvil-comp7 )"
+ fi
+fi
+
+mkdir -p "$ARTIFACTS"
+DONE_CANARY="$ARTIFACTS/conda-forge-build-done-${CONFIG}"
+rm -f "$DONE_CANARY"
+
+if [ -z "${CI}" ]; then
+ DOCKER_RUN_ARGS="-it "
+fi
+
+export UPLOAD_PACKAGES="${UPLOAD_PACKAGES:-True}"
+docker run ${DOCKER_RUN_ARGS} \
+ --shm-size=2G \
+ -v "${ARROW_ROOT}":/arrow:rw,z \
+ -e FEEDSTOCK_ROOT="/arrow/dev/tasks/conda-recipes" \
+ -e CONFIG \
+ -e HOST_USER_ID \
+ -e UPLOAD_PACKAGES \
+ -e ARROW_VERSION \
+ -e CI \
+ $DOCKER_IMAGE \
+ bash /arrow/dev/tasks/conda-recipes/build_steps.sh
+
+# verify that the end of the script was reached
+test -f "$DONE_CANARY"
diff --git a/dev/tasks/conda-recipes/travis.linux.yml b/dev/tasks/conda-recipes/travis.linux.yml
deleted file mode 100644
index 9e9f94f..0000000
--- a/dev/tasks/conda-recipes/travis.linux.yml
+++ /dev/null
@@ -1,78 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-os: linux
-dist: trusty
-language: generic
-
-# don't build twice
-if: tag IS blank
-
-env:
- global:
- - TRAVIS_TAG={{ task.tag }}
- - ARROW_VERSION={{ arrow.no_rc_version }}
- - PYTHONUNBUFFERED=1
-
-install:
- # Install Miniconda.
- - echo `pwd`
- - |
- echo ""
- echo "Installing a fresh version of Miniconda."
- MINICONDA_URL="https://repo.continuum.io/miniconda"
- MINICONDA_FILE="Miniconda3-latest-Linux-x86_64.sh"
- curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
- bash $MINICONDA_FILE -b
-
- # Install conda build dependency
- - |
- echo ""
- echo "Configuring conda."
- source /home/travis/miniconda3/bin/activate root
- conda install -n root -c conda-forge --quiet --yes conda-forge-ci-setup=2 conda-build
-
-before_script:
- - git clone --no-checkout {{ arrow.remote }} arrow
- - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
- - git -C arrow checkout FETCH_HEAD
-
- - pushd arrow/dev/tasks/conda-recipes
-
- # Configure conda
- - setup_conda_rc ./ ./ variants/{{ config }}.yaml
- - source run_conda_forge_build_setup
-
-script:
- - |
- conda build --croot $TRAVIS_HOME/conda_build_root \
- --output-folder . \
- -m variants/{{ config }}.yaml \
- parquet-cpp arrow-cpp pyarrow
-
-deploy:
- provider: releases
- api_key: $CROSSBOW_GITHUB_TOKEN
- file_glob: true
- file: $TRAVIS_BUILD_DIR/arrow/dev/tasks/conda-recipes/linux-64/*.tar.bz2
- skip_cleanup: true
- on:
- tags: true
-
-notifications:
- email:
- - {{ job.email }}
diff --git a/dev/tasks/conda-recipes/travis.osx.yml b/dev/tasks/conda-recipes/travis.osx.yml
deleted file mode 100644
index 3093cac..0000000
--- a/dev/tasks/conda-recipes/travis.osx.yml
+++ /dev/null
@@ -1,83 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-os: osx
-osx_image: xcode9.4
-language: generic
-
-# don't build twice
-if: tag IS blank
-
-env:
- global:
- - TRAVIS_TAG={{ task.tag }}
- - ARROW_VERSION={{ arrow.no_rc_version }}
- - PYTHONUNBUFFERED=1
-
-install:
- # Install Miniconda.
- - |
- echo ""
- echo "Installing a fresh version of Miniconda."
- MINICONDA_URL="https://repo.continuum.io/miniconda"
- MINICONDA_FILE="Miniconda3-latest-MacOSX-x86_64.sh"
- curl -L -O "${MINICONDA_URL}/${MINICONDA_FILE}"
- bash $MINICONDA_FILE -b
- # Install conda build dependency
- - |
- echo ""
- echo "Configuring conda."
- source /Users/travis/miniconda3/bin/activate root
- conda install -n root -c conda-forge --quiet --yes conda-forge-ci-setup=2 conda-build
-
-
-before_script:
- - git clone --no-checkout {{ arrow.remote }} arrow
- - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
- - git -C arrow checkout FETCH_HEAD
-
- - pushd arrow/dev/tasks/conda-recipes
-
- # Configure conda
- - setup_conda_rc ./ ./ variants/{{ config }}.yaml
-
- # XXX: workaround, see run_conda_forge_build_setup_osx#L33
- - mkdir -p ./.ci_support
- - cp variants/{{ config }}.yaml ./.ci_support/{{ config }}.yaml
- - CONFIG={{ config }} source run_conda_forge_build_setup
-
- # Compiler cleanup
- - mangle_compiler ./ ./ ./.ci_support/{{ config }}.yaml
-
-script:
- - |
- conda build --output-folder . \
- -m ./.ci_support/{{ config }}.yaml \
- parquet-cpp arrow-cpp pyarrow
-
-deploy:
- provider: releases
- api_key: $CROSSBOW_GITHUB_TOKEN
- file_glob: true
- file: $TRAVIS_BUILD_DIR/arrow/dev/tasks/conda-recipes/osx-64/*.tar.bz2
- skip_cleanup: true
- on:
- tags: true
-
-notifications:
- email:
- - {{ job.email }}
diff --git a/dev/tasks/crossbow.py b/dev/tasks/crossbow.py
index f7518d0..ab1b30b 100755
--- a/dev/tasks/crossbow.py
+++ b/dev/tasks/crossbow.py
@@ -20,18 +20,19 @@
import os
import re
import time
-import click
import hashlib
+from io import StringIO
+from pathlib import Path
+from textwrap import dedent
+from datetime import datetime
+from collections import namedtuple
+
+import click
import gnupg
import toolz
import pygit2
import github3
import jira.client
-
-from io import StringIO
-from pathlib import Path
-from textwrap import dedent
-from datetime import datetime
from jinja2 import Template, StrictUndefined
from setuptools_scm.git import parse as parse_git_version
from ruamel.yaml import YAML
@@ -73,6 +74,7 @@ def unflatten(mapping):
# - travis reads from the master branch and applies the rules
# - circle requires the configuration to be present on all branch, even ones
# that are configured to be skipped
+# - azure skips branches without azure-pipelines.yml by default
_default_travis_yml = """
branches:
@@ -226,6 +228,10 @@ class GitRemoteCallbacks(pygit2.RemoteCallbacks):
return None
+def _git_ssh_to_https(url):
+ return url.replace('git@github.com:', 'https://github.com/')
+
+
class Repo:
"""Base class for interaction with local git repositories
@@ -243,6 +249,7 @@ class Repo:
self.repo = pygit2.Repository(str(self.path))
self.github_token = github_token
self.require_https = require_https
+ self._github_repo = None # set by as_github_repo()
self._updated_refs = []
def __str__(self):
@@ -388,13 +395,14 @@ class Repo:
def as_github_repo(self):
"""Converts it to a repository object which wraps the GitHub API"""
- username, reponame = self._parse_github_user_repo()
- gh = github3.login(token=self.github_token)
- return gh.repository(username, reponame)
+ if self._github_repo is None:
+ username, reponame = self._parse_github_user_repo()
+ gh = github3.login(token=self.github_token)
+ return gh.repository(username, reponame)
+ return self._github_repo
-def _git_ssh_to_https(url):
- return url.replace('git@github.com:', 'https://github.com/')
+CombinedStatus = namedtuple('CombinedStatus', ('state', 'total_count'))
class Queue(Repo):
@@ -439,10 +447,71 @@ class Queue(Repo):
# create job's branch with its description
return self.create_branch(job.branch, files=job.render_files())
- def github_statuses(self, job):
+ def combined_status(self, task):
+ """Combine the results from status and checks API to a single state.
+
+ Azure pipelines uses checks API which doesn't provide a combined
+ interface like status API does, so we need to manually combine
+ both the commit statuses and the commit checks coming from
+ different API endpoint
+
+ Status.state: error, failure, pending or success, default pending
+ CheckRun.status: queued, in_progress or completed, default: queued
+ CheckRun.conclusion: success, failure, neutral, cancelled, timed_out
+ or action_required, only set if
+ CheckRun.status == 'completed'
+
+ 1. Convert CheckRun's status and conslusion to one of Status.state
+ 2. Merge the states based on the following rules:
+ - failure if any of the contexts report as error or failure
+ - pending if there are no statuses or a context is pending
+ - success if the latest status for all contexts is success
+ error otherwise.
+
+ Parameters
+ ----------
+ task : Task
+ Task to query the combined status for.
+
+ Returns
+ -------
+ combined_state: CombinedStatus(
+ state='error|failure|pending|success',
+ total_count='number of statuses and checks'
+ )
+ """
repo = self.as_github_repo()
- return {name: repo.commit(task.commit).status()
- for name, task in job.tasks.items()}
+ commit = repo.commit(task.commit)
+ states = []
+
+ for status in commit.status().statuses:
+ states.append(status.state)
+
+ for check in commit.check_runs():
+ if check.status == 'completed':
+ if check.conclusion in {'success', 'failure'}:
+ states.append(check.conclusion)
+ elif check.conclusion in {'cancelled', 'timed_out',
+ 'action_required'}:
+ states.append('error')
+ # omit `neutral` conslusion
+ else:
+ states.append('pending')
+
+ # it could be more effective, but the following is more descriptive
+ if any(state in {'error', 'failure'} for state in states):
+ combined_state = 'failure'
+ elif any(state == 'pending' for state in states):
+ combined_state = 'pending'
+ elif all(state == 'success' for state in states):
+ combined_state = 'success'
+ else:
+ combined_state = 'error'
+
+ return CombinedStatus(state=combined_state, total_count=len(states))
+
+ def github_statuses(self, job):
+ return toolz.valmap(self.combined_status, job.tasks)
def github_assets(self, task):
repo = self.as_github_repo()
@@ -548,7 +617,7 @@ class Task:
def __init__(self, platform, ci, template, artifacts=None, params=None):
assert platform in {'win', 'osx', 'linux'}
- assert ci in {'circle', 'travis', 'appveyor'}
+ assert ci in {'circle', 'travis', 'appveyor', 'azure'}
self.ci = ci
self.platform = platform
self.template = template
@@ -573,7 +642,8 @@ class Task:
config_files = {
'circle': '.circleci/config.yml',
'travis': '.travis.yml',
- 'appveyor': 'appveyor.yml'
+ 'appveyor': 'appveyor.yml',
+ 'azure': 'azure-pipelines.yml'
}
return config_files[self.ci]
diff --git a/dev/tasks/conda-recipes/pyarrow/build.sh b/dev/tasks/docker-tests/azure.linux.yml
similarity index 54%
copy from dev/tasks/conda-recipes/pyarrow/build.sh
copy to dev/tasks/docker-tests/azure.linux.yml
index 56d6fa7..8d54b69 100644
--- a/dev/tasks/conda-recipes/pyarrow/build.sh
+++ b/dev/tasks/docker-tests/azure.linux.yml
@@ -1,5 +1,3 @@
-#!/bin/sh
-
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -17,21 +15,31 @@
# specific language governing permissions and limitations
# under the License.
-set -e
-set -x
+jobs:
+- job: linux
+ pool:
+ vmImage: ubuntu-16.04
+ timeoutInMinutes: 360
-# Build dependencies
-export ARROW_HOME=$PREFIX
-export PARQUET_HOME=$PREFIX
-export SETUPTOOLS_SCM_PRETEND_VERSION=$PKG_VERSION
+ steps:
+ - task: DockerInstaller@0
+ displayName: Docker Installer
+ inputs:
+ dockerVersion: 17.09.0-ce
+ releaseType: stable
-cd python
+ - script: |
+ git clone --no-checkout {{ arrow.remote }} arrow
+ git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ git -C arrow checkout FETCH_HEAD
+ git -C arrow submodule update --init --recursive
+ displayName: Clone arrow
-$PYTHON setup.py \
- build_ext --build-type=release \
- --with-orc \
- --with-plasma \
- --with-parquet \
- --with-gandiva \
- install --single-version-externally-managed \
- --record=record.txt
+ - script: |
+ pushd arrow
+ {%- for command in commands %}
+ {{ command }}
+ {%- endfor %}
+ popd
+ workingDirectory: arrow
+ displayName: Run docker test
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 875216c..9030aef 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -72,9 +72,9 @@ tasks:
############################## Conda Linux ##################################
conda-linux-gcc-py27:
- ci: travis
+ ci: azure
platform: linux
- template: conda-recipes/travis.linux.yml
+ template: conda-recipes/azure.linux.yml
params:
config: linux_python2.7
artifacts:
@@ -82,9 +82,9 @@ tasks:
- pyarrow-{no_rc_version}-py27(h[a-z0-9]+)_0.tar.bz2
conda-linux-gcc-py36:
- ci: travis
+ ci: azure
platform: linux
- template: conda-recipes/travis.linux.yml
+ template: conda-recipes/azure.linux.yml
params:
config: linux_python3.6
artifacts:
@@ -92,9 +92,9 @@ tasks:
- pyarrow-{no_rc_version}-py36(h[a-z0-9]+)_0.tar.bz2
conda-linux-gcc-py37:
- ci: travis
+ ci: azure
platform: linux
- template: conda-recipes/travis.linux.yml
+ template: conda-recipes/azure.linux.yml
params:
config: linux_python3.7
artifacts:
@@ -104,9 +104,9 @@ tasks:
############################## Conda OSX ####################################
conda-osx-clang-py27:
- ci: travis
+ ci: azure
platform: osx
- template: conda-recipes/travis.osx.yml
+ template: conda-recipes/azure.osx.yml
params:
config: osx_python2.7
artifacts:
@@ -114,9 +114,9 @@ tasks:
- pyarrow-{no_rc_version}-py27(h[a-z0-9]+)_0.tar.bz2
conda-osx-clang-py36:
- ci: travis
+ ci: azure
platform: osx
- template: conda-recipes/travis.osx.yml
+ template: conda-recipes/azure.osx.yml
params:
config: osx_python3.6
artifacts:
@@ -124,8 +124,9 @@ tasks:
- pyarrow-{no_rc_version}-py36(h[a-z0-9]+)_0.tar.bz2
conda-osx-clang-py37:
+ ci: azure
platform: osx
- template: conda-recipes/travis.osx.yml
+ template: conda-recipes/azure.osx.yml
params:
config: osx_python3.7
artifacts:
@@ -137,21 +138,21 @@ tasks:
conda-win-vs2015-py36:
ci: appveyor
platform: win
- template: conda-recipes/appveyor.yml
+ template: conda-recipes/appveyor.win.yml
params:
- config: win_c_compilervs2015cxx_compilervs2015python3.6vc14
+ config: win_c_compilervs2015cxx_compilervs2015python3.6
artifacts:
- - arrow-cpp-{no_rc_version}-py36_vc14(h[a-z0-9]+)_0.tar.bz2
+ - arrow-cpp-{no_rc_version}-py36(h[a-z0-9]+)_0.tar.bz2
- pyarrow-{no_rc_version}-py36(h[a-z0-9]+)_0.tar.bz2
conda-win-vs2015-py37:
ci: appveyor
platform: win
- template: conda-recipes/appveyor.yml
+ template: conda-recipes/appveyor.win.yml
params:
- config: win_c_compilervs2015cxx_compilervs2015python3.7vc14
+ config: win_c_compilervs2015cxx_compilervs2015python3.7
artifacts:
- - arrow-cpp-{no_rc_version}-py37_vc14(h[a-z0-9]+)_0.tar.bz2
+ - arrow-cpp-{no_rc_version}-py37(h[a-z0-9]+)_0.tar.bz2
- pyarrow-{no_rc_version}-py37(h[a-z0-9]+)_0.tar.bz2
############################## Wheel Linux ##################################
diff --git a/dev/tasks/upload-assets.py b/dev/tasks/upload-assets.py
new file mode 100755
index 0000000..2435633
--- /dev/null
+++ b/dev/tasks/upload-assets.py
@@ -0,0 +1,64 @@
+#!/usr/bin/env python
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import os
+import mimetypes
+from glob import glob
+
+import click
+import github3
+
+
+# TODO(kszucs): move it to crossbow.py
+
+
+@click.command()
+@click.option('--sha', help='Target committish')
+@click.option('--tag', help='Target tag')
+@click.option('--pattern', help='File pattern')
+def upload_assets(tag, sha, pattern):
+ token = os.environ['CROSSBOW_GITHUB_TOKEN']
+ owner, repository = os.environ['CROSSBOW_GITHUB_REPO'].split('/')
+
+ gh = github3.login(token=token)
+ repo = gh.repository(owner, repository)
+ click.echo('Selected repository: {}/{}'.format(owner, repository))
+
+ try:
+ release = repo.release_from_tag(tag)
+ except github3.exceptions.NotFoundError:
+ pass
+ else:
+ click.echo('Removing release `{}`'.format(release.tag_name))
+ release.delete()
+
+ click.echo('Creating release `{}`'.format(tag))
+ release = repo.create_release(tag, sha)
+
+ for path in glob(pattern):
+ name = os.path.basename(path)
+ mime = mimetypes.guess_type(name)[0] or 'application/octet-stream'
+
+ click.echo('Uploading asset `{}`...'.format(name))
+ with open(path, 'rb') as fp:
+ release.upload_asset(name=name, asset=fp, content_type=mime)
+
+
+if __name__ == '__main__':
+ upload_assets()