You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/04 18:25:39 UTC
[GitHub] [arrow] kszucs opened a new pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
kszucs opened a new pull request #9096:
URL: https://github.com/apache/arrow/pull/9096
Main features:
- dockerize windows builds
- use vcpkg as the dependency source
Going to write a more detailed description including commands to try out with.
cc @kou @xhochy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs removed a comment on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs removed a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-760317968
@github-actions crossbow -c ursacomputing/crossbow submit -g packaging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553880289
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
I plan to implement this bundling logic in cmake, but we can defer it for after the release.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
We can choose to bundle security sensitive shared libraries instead of linking them statically.
##########
File path: dev/tasks/tasks.yml
##########
@@ -586,29 +490,31 @@ tasks:
############################## Wheel Windows ################################
- wheel-win-cp36m:
+ wheel-windows-cp36m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.6
artifacts:
- pyarrow-{no_rc_version}-cp36-cp36m-win_amd64.whl
- wheel-win-cp37m:
+ wheel-windows-cp37m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.7
artifacts:
- pyarrow-{no_rc_version}-cp37-cp37m-win_amd64.whl
- wheel-win-cp38:
+ wheel-windows-cp38m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.8
artifacts:
- - pyarrow-{no_rc_version}-cp38-cp38-win_amd64.whl
+ - pyarrow-{no_rc_version}-cp38-cp38m-win_amd64.whl
+
+ # TODO(kszucs): add python 3.9
Review comment:
`choco` fails to install python 3.9, investigating it.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
> In the case of OpenSSL, if we bundled a shared version the version in the main system will be preferred.
Shouldn't we prefer the system one over the bundled one?
> Not sure though where this is updated from.
You mean the bundled or the system one?
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
How about keeping it as is, linking these libraries statically?
##########
File path: dev/tasks/tasks.yml
##########
@@ -586,29 +490,31 @@ tasks:
############################## Wheel Windows ################################
- wheel-win-cp36m:
+ wheel-windows-cp36m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.6
artifacts:
- pyarrow-{no_rc_version}-cp36-cp36m-win_amd64.whl
- wheel-win-cp37m:
+ wheel-windows-cp37m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.7
artifacts:
- pyarrow-{no_rc_version}-cp37-cp37m-win_amd64.whl
- wheel-win-cp38:
+ wheel-windows-cp38m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.8
artifacts:
- - pyarrow-{no_rc_version}-cp38-cp38-win_amd64.whl
+ - pyarrow-{no_rc_version}-cp38-cp38m-win_amd64.whl
+
+ # TODO(kszucs): add python 3.9
Review comment:
Fixed.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
Ok, I'm going to clean up the commented section used for bundling from `python/CMakeLists.txt` and `setup.py` then.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
And create a follow-up JIRA.
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
Created a follow-up jira https://issues.apache.org/jira/browse/ARROW-11185
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
https://issues.apache.org/jira/browse/ARROW-11186
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-760435612
@kou this should be ready to merge. One of the arm linux builds is [failing with a timeout error](https://github.com/ursacomputing/crossbow/runs/1704000375) in addition to the arm conda drone builds, but these shouldn't block this PR. The latest crossbow submission is here https://github.com/ursacomputing/crossbow/branches/all?query=build-18
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756357264
Revision: 233aabae02f16ef8edf55b5470cb7ab12c58c920
Submitted crossbow builds: [ursa-labs/crossbow @ actions-838](https://github.com/ursa-labs/crossbow/branches/all?query=actions-838)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-838-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-838-github-wheel-manylinux2014-cp36m)|
|wheel-windows-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-838-github-wheel-windows-cp36m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r555070312
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
Review comment:
Updated.
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758037754
Revision: 45d8bc3f452a6068114319bc83b6c3fcd55473ab
Submitted crossbow builds: [ursa-labs/crossbow @ actions-869](https://github.com/ursa-labs/crossbow/branches/all?query=actions-869)
|Task|Status|
|----|------|
|wheel-osx-high-sierra-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-869-github-wheel-osx-high-sierra-cp36m)|
|wheel-osx-mavericks-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-869-github-wheel-osx-mavericks-cp36m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756746930
Revision: b47406064789942bacaea9d3b1d62d5778796d37
Submitted crossbow builds: [ursa-labs/crossbow @ actions-844](https://github.com/ursa-labs/crossbow/branches/all?query=actions-844)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-844-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-844-github-wheel-manylinux2014-cp36m)|
|wheel-windows-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-844-github-wheel-windows-cp36m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759830041
@pitrou you have resolved it previously with https://github.com/pitrou/arrow/commit/b0dd66f17e94fbf61da0f1f616ccd5e627a79145. Shall we patch the unix portfile to include https://github.com/pitrou/arrow/commit/b0dd66f17e94fbf61da0f1f616ccd5e627a79145#diff-7ac648d4cfa17e043ddbdccf78bcfacbca24e26020af1a56e0fbfef2f8e9aa1eR43-R44 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754868233
Currently integrating with crossbow using pre-built images, the manylinux build already works except the artifact uploading.
Pull & build took ~13m: https://github.com/ursa-labs/crossbow/runs/1652174767
Passing build under ~19m: https://github.com/ursa-labs/crossbow/runs/1652276423
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756369295
@kou @xhochy bundled the msvc runtime. Locally tested with the stock windows python images and seems to work fine. Hopefully crossbow will confirm it.
I'm going to write a more detailed description of the changes tomorrow. Any reviews are welcome!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kou closed pull request #9096:
URL: https://github.com/apache/arrow/pull/9096
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-760317968
@github-actions crossbow -c ursacomputing/crossbow submit -g packaging
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759830041
@pitrou you have resolved it previously with https://github.com/pitrou/arrow/commit/b0dd66f17e94fbf61da0f1f616ccd5e627a79145
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-755716186
@xhochy we have a [running build](https://github.com/ursa-labs/crossbow/runs/1658737075) for windows on github actions now. I'll do a couple of things to try to improve the build times a bit (make the image smaller because pulling takes ~15m and enable unity builds).
Since we don't bundle `msvcp140.dll` these wheels are not working in the official windows python docker containers, so I'm using the same image as for building. I may try to bundle it (guess it should be about copying it into the archive) to ensure that it works in the base images.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756800569
@github-actions crossbow submit wheel-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou edited a comment on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
pitrou edited a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759535264
@kszucs We would need to apply the following patch when building snappy in order to solve ARROW-7939. Is it possible?
https://github.com/conda-forge/snappy-feedstock/blob/5b7b850667c769df4551c81dc8ffaef76e8d371b/recipe/disable_bmi.patch
Edit: done, I hope that's ok.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs removed a comment on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs removed a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754881637
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756821064
Revision: 8081526709fc1642d0de3c305e4bb7cfa8af6912
Submitted crossbow builds: [ursa-labs/crossbow @ actions-845](https://github.com/ursa-labs/crossbow/branches/all?query=actions-845)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2010-cp38)|
|wheel-manylinux2010-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2010-cp39)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2014-cp38)|
|wheel-manylinux2014-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-manylinux2014-cp39)|
|wheel-osx-high-sierra-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-high-sierra-cp36m)|
|wheel-osx-high-sierra-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-high-sierra-cp37m)|
|wheel-osx-high-sierra-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-high-sierra-cp38)|
|wheel-osx-high-sierra-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-high-sierra-cp39)|
|wheel-osx-mavericks-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-mavericks-cp36m)|
|wheel-osx-mavericks-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-mavericks-cp37m)|
|wheel-osx-mavericks-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-mavericks-cp38)|
|wheel-osx-mavericks-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-osx-mavericks-cp39)|
|wheel-windows-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-windows-cp36m)|
|wheel-windows-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-windows-cp37m)|
|wheel-windows-cp38m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-windows-cp38m)|
|wheel-windows-cp39m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-845-github-wheel-windows-cp39m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs removed a comment on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs removed a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758282789
@github-actions crossbow -c ursacomputing/crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759732589
Revision: 8a9a2193079d5251636395767f0757ec68eb8289
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2010-cp38)|
|wheel-manylinux2010-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2010-cp39)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2014-cp38)|
|wheel-manylinux2014-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-manylinux2014-cp39)|
|wheel-osx-high-sierra-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-high-sierra-cp36m)|
|wheel-osx-high-sierra-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-high-sierra-cp37m)|
|wheel-osx-high-sierra-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-high-sierra-cp38)|
|wheel-osx-high-sierra-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-high-sierra-cp39)|
|wheel-osx-mavericks-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-mavericks-cp36m)|
|wheel-osx-mavericks-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-mavericks-cp37m)|
|wheel-osx-mavericks-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-mavericks-cp38)|
|wheel-osx-mavericks-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-osx-mavericks-cp39)|
|wheel-windows-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-windows-cp36m)|
|wheel-windows-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-windows-cp37m)|
|wheel-windows-cp38m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-windows-cp38m)|
|wheel-windows-cp39m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-2-github-wheel-windows-cp39m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756418757
It seems that https://github.com/ursa-labs/crossbow/releases/tag/actions-838-github-wheel-windows-cp36m doesn't have artifacts...
Where did you download built wheels?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553983800
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
https://issues.apache.org/jira/browse/ARROW-11186
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556020595
##########
File path: python/pyarrow/tests/test_fs.py
##########
@@ -1326,7 +1326,10 @@ def test_py_open_append_stream():
f.write(b"data")
+# TODO(kszucs): the following two cases fail inside non-default
+# AWS regions e.g. us-east-2
@pytest.mark.s3
+@pytest.mark.skip
Review comment:
Nice catch, definitely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758282789
@github-actions crossbow -c ursacomputing/crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r555070445
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
Review comment:
Updated.
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2014_x86_64?tab=tags
+ python-wheel-manylinux-2014:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ build:
+ args:
+ base: quay.io/pypa/manylinux2014_x86_64:2020-11-11-bc8ce45
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2014
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2014-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ python-wheel-manylinux-test:
+ image: python:${PYTHON}
+ shm_size: 2G
+ volumes:
+ - .:/arrow:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_test.sh
+
+ python-wheel-windows-vs2017:
+ image: ${REPO}:python-${PYTHON}-wheel-windows-vs2017
+ build:
+ args:
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-windows-vs2017.dockerfile
+ # This should make the pushed images reusable, but the image gets rebuilt
+ # no matter
+ # cache_from:
+ # - mcr.microsoft.com/windows/servercore:ltsc2019
+ # - ${REPO}:wheel-windows-vs2017
+ volumes:
+ - "${DOCKER_VOLUME_PREFIX}wheel-windows-clcache:C:/clcache"
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_build.bat
+
+ python-wheel-windows-test:
+ image: python:${PYTHON}-windowsservercore-1809
+ volumes:
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_test.bat
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759533582
I'm building the images and submitting a crossbow job afterwards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r555358083
##########
File path: .env
##########
@@ -18,6 +18,12 @@
# All of the following environment variables are required to set default values
# for the parameters in docker-compose.yml.
+# empty prefix means bind mounted volumes which prevents a number
+# of permission issues
+DOCKER_VOLUME_PREFIX=""
+# should setup this on GHA to keep the cache plugin functional
+# DOCKER_VOLUME_PREFIX=".docker/"
Review comment:
Need to set this in all the github actions workfloww where docker-compose is used to keep the cache plugin functional.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553936887
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
Not sure about this. I don't know where the system one is coming from.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-760549176
Thanks!
I'll merge this. We need to work on some follow-up tasks:
* Fix .deb/.rpm builds on Travis CI
* Fix Python version (3.8) in `dev/tasks/conda-recipes/.ci_support/osx_pytho3.7.____cpython.yaml`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553955906
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
How about keeping it as is, linking these libraries statically?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759808670
Hmm, the SSL certificate check failure on manylinux seems serious and should be fixed. It may be due to this line:
https://github.com/microsoft/vcpkg/blob/master/ports/openssl/unix/CMakeLists.txt#L171
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759816099
Also, looking at https://github.com/microsoft/vcpkg/blob/master/ports/openssl/unix/CMakeLists.txt#L8-L13, it does not seem Linux builds would be supported on non-x86 platforms (such as arm64).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556914854
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -2466,7 +2470,7 @@ macro(build_grpc)
-DgRPC_GFLAGS_PROVIDER=package
-DgRPC_MSVC_STATIC_RUNTIME=${ARROW_USE_STATIC_CRT}
-DgRPC_PROTOBUF_PROVIDER=package
- -DgRPC_PROTOBUF_PACKAGE_TYPE=CONFIG
+ -DgRPC_PROTOBUF_PACKAGE_TYPE=${gRPC_PROTOBUF_PACKAGE_TYPE}
Review comment:
How about the following?
```diff
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index e5c72be89..1ebf836a3 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -1326,6 +1326,7 @@ endif()
macro(build_protobuf)
message("Building Protocol Buffers from source")
+ set(PROTOBUF_VENDORED TRUE)
set(PROTOBUF_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/protobuf_ep-install")
set(PROTOBUF_INCLUDE_DIR "${PROTOBUF_PREFIX}/include")
# Newer protobuf releases always have a lib prefix independent from CMAKE_STATIC_LIBRARY_PREFIX
@@ -2466,13 +2467,15 @@ macro(build_grpc)
-DgRPC_GFLAGS_PROVIDER=package
-DgRPC_MSVC_STATIC_RUNTIME=${ARROW_USE_STATIC_CRT}
-DgRPC_PROTOBUF_PROVIDER=package
- -DgRPC_PROTOBUF_PACKAGE_TYPE=CONFIG
-DgRPC_RE2_PROVIDER=package
-DgRPC_SSL_PROVIDER=package
-DgRPC_ZLIB_PROVIDER=package
-DCMAKE_INSTALL_PREFIX=${GRPC_PREFIX}
-DCMAKE_INSTALL_LIBDIR=lib
-DBUILD_SHARED_LIBS=OFF)
+ if(PROTOBUF_VENDORED)
+ list(APPEND GRPC_CMAKE_ARGS -DgRPC_PROTOBUF_PACKAGE_TYPE=CONFIG)
+ endif()
if(OPENSSL_ROOT_DIR)
list(APPEND GRPC_CMAKE_ARGS -DOPENSSL_ROOT_DIR=${OPENSSL_ROOT_DIR})
endif()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553924472
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
We can choose to bundle security sensitive shared libraries instead of linking them statically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758265696
@github-actions crossbow -c ursacomputing/crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-760363251
I submitted the crossbow tasks using the CLI, the in-progress results are promising:
https://github.com/ursacomputing/crossbow/branches/all?query=build-17
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553935735
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
> In the case of OpenSSL, if we bundled a shared version the version in the main system will be preferred.
Shouldn't we prefer the system one over the bundled one?
> Not sure though where this is updated from.
You mean the bundled or the system one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756449204
@github-actions crossbow submit wheel-windows-cp36m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754886364
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756356452
@github-actions crossbow submit wheel-manylinux2010-cp36m wheel-manylinux2014-cp36m wheel-windows-cp36m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553972594
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
And create a follow-up JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756449886
Revision: 009d9d049ef48e3cd6e4e17126d6093b25331312
Submitted crossbow builds: [ursa-labs/crossbow @ actions-842](https://github.com/ursa-labs/crossbow/branches/all?query=actions-842)
|Task|Status|
|----|------|
|wheel-windows-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-842-github-wheel-windows-cp36m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754868233
Currently integrating with crossbow using pre-built images, the manylinux build already works except the artifact uploading.
Pull & build took ~13m: https://github.com/ursa-labs/crossbow/runs/1652174767
Passing build under ~19m including testing: https://github.com/ursa-labs/crossbow/runs/1652276423
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756746930
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756917261
The osx builds shouldn't be failing, but that's out of the scope of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556910519
##########
File path: dev/tasks/python-wheels/osx-build.sh
##########
@@ -91,6 +92,7 @@ function build_wheel {
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DMAKE=make \
-DProtobuf_SOURCE=SYSTEM \
+ -DgRPC_PROTOBUF_PACKAGE_TYPE=SYSTEM \
Review comment:
I think that this is needless.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553926908
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
In the case of OpenSSL, if we bundled a shared version the version in the main system will be preferred. Not sure though where this is updated from. In `conda-forge` we explicitly have a Python setup that takes care that one is not used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553880289
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
I plan to implement this bundling logic in cmake, but we can defer it for after the release.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556474284
##########
File path: ci/scripts/python_wheel_manylinux_build.sh
##########
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -ex
+
+function check_arrow_visibility {
+ nm --demangle --dynamic /tmp/arrow-dist/lib/libarrow.so > nm_arrow.log
+
+ # Filter out Arrow symbols and see if anything remains.
+ # '_init' and '_fini' symbols may or not be present, we don't care.
+ # (note we must ignore the grep exit status when no match is found)
+ grep ' T ' nm_arrow.log | grep -v -E '(arrow|\b_init\b|\b_fini\b)' | cat - > visible_symbols.log
+
+ if [[ -f visible_symbols.log && `cat visible_symbols.log | wc -l` -eq 0 ]]; then
+ return 0
+ else
+ echo "== Unexpected symbols exported by libarrow.so =="
+ cat visible_symbols.log
+ echo "================================================"
+
+ exit 1
+ fi
+}
+
+echo "=== (${PYTHON_VERSION}) Clear output directories and leftovers ==="
+# Clear output directories and leftovers
+rm -rf /tmp/arrow-build
+rm -rf /arrow/python/dist
+rm -rf /arrow/python/build
+rm -rf /arrow/python/repaired_wheels
+rm -rf /arrow/python/pyarrow/*.so
+rm -rf /arrow/python/pyarrow/*.so.*
+
+echo "=== (${PYTHON_VERSION}) Building Arrow C++ libraries ==="
+: ${ARROW_DATASET:=ON}
+: ${ARROW_FLIGHT:=ON}
+: ${ARROW_GANDIVA:=OFF}
+: ${ARROW_HDFS:=ON}
+: ${ARROW_JEMALLOC:=ON}
+: ${ARROW_MIMALLOC:=ON}
+: ${ARROW_ORC:=ON}
+: ${ARROW_PARQUET:=ON}
+: ${ARROW_PLASMA:=ON}
+: ${ARROW_S3:=ON}
+: ${ARROW_TENSORFLOW:=ON}
+: ${ARROW_WITH_BROTLI:=ON}
+: ${ARROW_WITH_BZ2:=ON}
+: ${ARROW_WITH_LZ4:=ON}
+: ${ARROW_WITH_SNAPPY:=ON}
+: ${ARROW_WITH_ZLIB:=ON}
+: ${ARROW_WITH_ZSTD:=ON}
+: ${CMAKE_BUILD_TYPE:=release}
+: ${CMAKE_GENERATOR:=Ninja}
+
+mkdir /tmp/arrow-build
+pushd /tmp/arrow-build
+cmake \
+ -DARROW_BROTLI_USE_SHARED=OFF \
+ -DARROW_BUILD_SHARED=ON \
+ -DARROW_BUILD_STATIC=OFF \
+ -DARROW_BUILD_TESTS=OFF \
+ -DARROW_DATASET=${ARROW_DATASET} \
+ -DARROW_DEPENDENCY_SOURCE="SYSTEM" \
+ -DARROW_DEPENDENCY_USE_SHARED=OFF \
+ -DARROW_FLIGHT==${ARROW_FLIGHT} \
+ -DARROW_GANDIVA=${ARROW_GANDIVA} \
+ -DARROW_HDFS=${ARROW_HDFS} \
+ -DARROW_JEMALLOC=${ARROW_JEMALLOC} \
+ -DARROW_MIMALLOC=${ARROW_MIMALLOC} \
+ -DARROW_ORC=${ARROW_ORC} \
+ -DARROW_PACKAGE_KIND="manylinux${MANYLINUX_VERSION}" \
+ -DARROW_PARQUET=${ARROW_PARQUET} \
+ -DARROW_PLASMA=${ARROW_PLASMA} \
+ -DARROW_PYTHON=ON \
+ -DARROW_RPATH_ORIGIN=ON \
+ -DARROW_S3=${ARROW_S3} \
+ -DARROW_TENSORFLOW=${ARROW_TENSORFLOW} \
+ -DARROW_UTF8PROC_USE_SHARED=OFF \
+ -DARROW_WITH_BROTLI=${ARROW_WITH_BROTLI} \
+ -DARROW_WITH_BZ2=${ARROW_WITH_BZ2} \
+ -DARROW_WITH_LZ4=${ARROW_WITH_LZ4} \
+ -DARROW_WITH_SNAPPY=${ARROW_WITH_SNAPPY} \
+ -DARROW_WITH_ZLIB=${ARROW_WITH_ZLIB} \
+ -DARROW_WITH_ZSTD=${ARROW_WITH_ZSTD} \
+ -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
+ -DCMAKE_INSTALL_LIBDIR=lib \
+ -DCMAKE_INSTALL_PREFIX=/tmp/arrow-dist \
+ -DCMAKE_TOOLCHAIN_FILE=/opt/vcpkg/scripts/buildsystems/vcpkg.cmake \
+ -DCMAKE_UNITY_BUILD=OFF \
Review comment:
Switch to ON to make builds slightly faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756989524
The OSX issue is that `protobuf` is installed through `brew` and that doesn't install it using `cmake`. You could conditionally add the following line https://github.com/apache/arrow/blob/a576cd71c08c3fec029d24092c407ece1775f5a6/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2459 only when protobuf is also build using bundled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r555358083
##########
File path: .env
##########
@@ -18,6 +18,12 @@
# All of the following environment variables are required to set default values
# for the parameters in docker-compose.yml.
+# empty prefix means bind mounted volumes which prevents a number
+# of permission issues
+DOCKER_VOLUME_PREFIX=""
+# should setup this on GHA to keep the cache plugin functional
+# DOCKER_VOLUME_PREFIX=".docker/"
Review comment:
Need to set this in the github actions workflow where docker-compose is used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756738047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553621162
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
We can detect redist version (`14.16.27012` in this command line) by reading content at `C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\Microsoft.VCRedistVersion.default.txt`.
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
I think that we also need to bundle `vcruntime140.dll` that is depended by `msvcp140.dll`. It's located at the same directory of `msvcp140.dll`: `C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\vcruntime140.dll`
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
Ah, Python for Windows includes `vcruntime140.dll`.
We don't need to bundle them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754900016
Revision: aeb902f4f50e70fb6be844ecaf1283820bee3c64
Submitted crossbow builds: [ursa-labs/crossbow @ actions-825](https://github.com/ursa-labs/crossbow/branches/all?query=actions-825)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2010-cp38)|
|wheel-manylinux2010-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2010-cp39)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2014-cp38)|
|wheel-manylinux2014-cp39|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-825-github-wheel-manylinux2014-cp39)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754881637
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756449739
The build also executes the tests in a different docker container, see the "Test Wheel" step in the [build](https://github.com/ursa-labs/crossbow/runs/1664830035?check_suite_focus=true)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758698814
Revision: 3eb191c38d0ef1b9f7af9f264e80e9364913de1a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-0](https://github.com/ursacomputing/crossbow/branches/all?query=actions-0)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2010-cp38)|
|wheel-manylinux2010-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2010-cp39)|
|wheel-manylinux2014-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2014-cp38)|
|wheel-manylinux2014-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-manylinux2014-cp39)|
|wheel-osx-high-sierra-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-high-sierra-cp36m)|
|wheel-osx-high-sierra-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-high-sierra-cp37m)|
|wheel-osx-high-sierra-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-high-sierra-cp38)|
|wheel-osx-high-sierra-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-high-sierra-cp39)|
|wheel-osx-mavericks-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-mavericks-cp36m)|
|wheel-osx-mavericks-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-mavericks-cp37m)|
|wheel-osx-mavericks-cp38|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-mavericks-cp38)|
|wheel-osx-mavericks-cp39|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-osx-mavericks-cp39)|
|wheel-windows-cp36m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-windows-cp36m)|
|wheel-windows-cp37m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-windows-cp37m)|
|wheel-windows-cp38m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-windows-cp38m)|
|wheel-windows-cp39m|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-0-github-wheel-windows-cp39m)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759732082
@github-actions crossbow -c ursacomputing/crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556002640
##########
File path: python/pyarrow/tests/test_fs.py
##########
@@ -1326,7 +1326,10 @@ def test_py_open_append_stream():
f.write(b"data")
+# TODO(kszucs): the following two cases fail inside non-default
+# AWS regions e.g. us-east-2
@pytest.mark.s3
+@pytest.mark.skip
Review comment:
Should this be reverted before merging?
(I thought @xhochy encountered a similar some time ago?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-757974715
@github-actions crossbow submit wheel-osx-mavericks-cp36m wheel-osx-high-sierra-cp36m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553971840
##########
File path: dev/tasks/tasks.yml
##########
@@ -586,29 +490,31 @@ tasks:
############################## Wheel Windows ################################
- wheel-win-cp36m:
+ wheel-windows-cp36m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.6
artifacts:
- pyarrow-{no_rc_version}-cp36-cp36m-win_amd64.whl
- wheel-win-cp37m:
+ wheel-windows-cp37m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.7
artifacts:
- pyarrow-{no_rc_version}-cp37-cp37m-win_amd64.whl
- wheel-win-cp38:
+ wheel-windows-cp38m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.8
artifacts:
- - pyarrow-{no_rc_version}-cp38-cp38-win_amd64.whl
+ - pyarrow-{no_rc_version}-cp38-cp38m-win_amd64.whl
+
+ # TODO(kszucs): add python 3.9
Review comment:
Fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r554240959
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
Review comment:
How about prepend `python-` because `python-` exists in related names?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
Review comment:
How about prepend `python-` because `python-` exists in related names?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2014_x86_64?tab=tags
+ python-wheel-manylinux-2014:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ build:
+ args:
+ base: quay.io/pypa/manylinux2014_x86_64:2020-11-11-bc8ce45
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2014
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2014-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ python-wheel-manylinux-test:
+ image: python:${PYTHON}
+ shm_size: 2G
+ volumes:
+ - .:/arrow:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_test.sh
+
+ python-wheel-windows-vs2017:
+ image: ${REPO}:python-${PYTHON}-wheel-windows-vs2017
+ build:
+ args:
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-windows-vs2017.dockerfile
+ # This should make the pushed images reusable, but the image gets rebuilt
+ # no matter
+ # cache_from:
+ # - mcr.microsoft.com/windows/servercore:ltsc2019
+ # - ${REPO}:wheel-windows-vs2017
+ volumes:
+ - "${DOCKER_VOLUME_PREFIX}wheel-windows-clcache:C:/clcache"
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_build.bat
+
+ python-wheel-windows-test:
+ image: python:${PYTHON}-windowsservercore-1809
+ volumes:
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_test.bat
Review comment:
How about moving `python-wheel-*` services into `...### Python ###...` section?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
Review comment:
How about prepend `python_` because `python-` exists in related names?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754141802
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754868233
Currently integrating with crossbow using pre-built images, the manylinux build already works except the artifact uploading.
Pull & build took ~13m: https://github.com/ursa-labs/crossbow/runs/1652174767
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-759535264
@kszucs We would need to apply the following patch when building snappy in order to solve ARROW-7939. Is it possible?
https://github.com/conda-forge/snappy-feedstock/blob/5b7b850667c769df4551c81dc8ffaef76e8d371b/recipe/disable_bmi.patch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553924887
##########
File path: dev/tasks/tasks.yml
##########
@@ -586,29 +490,31 @@ tasks:
############################## Wheel Windows ################################
- wheel-win-cp36m:
+ wheel-windows-cp36m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.6
artifacts:
- pyarrow-{no_rc_version}-cp36-cp36m-win_amd64.whl
- wheel-win-cp37m:
+ wheel-windows-cp37m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.7
artifacts:
- pyarrow-{no_rc_version}-cp37-cp37m-win_amd64.whl
- wheel-win-cp38:
+ wheel-windows-cp38m:
ci: github
- template: python-wheels/github.win.yml
+ template: python-wheels/github.windows.yml
params:
python_version: 3.8
artifacts:
- - pyarrow-{no_rc_version}-cp38-cp38-win_amd64.whl
+ - pyarrow-{no_rc_version}-cp38-cp38m-win_amd64.whl
+
+ # TODO(kszucs): add python 3.9
Review comment:
`choco` fails to install python 3.9, investigating it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r556477309
##########
File path: ci/scripts/python_wheel_manylinux_build.sh
##########
@@ -0,0 +1,141 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -ex
+
+function check_arrow_visibility {
+ nm --demangle --dynamic /tmp/arrow-dist/lib/libarrow.so > nm_arrow.log
+
+ # Filter out Arrow symbols and see if anything remains.
+ # '_init' and '_fini' symbols may or not be present, we don't care.
+ # (note we must ignore the grep exit status when no match is found)
+ grep ' T ' nm_arrow.log | grep -v -E '(arrow|\b_init\b|\b_fini\b)' | cat - > visible_symbols.log
+
+ if [[ -f visible_symbols.log && `cat visible_symbols.log | wc -l` -eq 0 ]]; then
+ return 0
+ else
+ echo "== Unexpected symbols exported by libarrow.so =="
+ cat visible_symbols.log
+ echo "================================================"
+
+ exit 1
+ fi
+}
+
+echo "=== (${PYTHON_VERSION}) Clear output directories and leftovers ==="
+# Clear output directories and leftovers
+rm -rf /tmp/arrow-build
+rm -rf /arrow/python/dist
+rm -rf /arrow/python/build
+rm -rf /arrow/python/repaired_wheels
+rm -rf /arrow/python/pyarrow/*.so
+rm -rf /arrow/python/pyarrow/*.so.*
+
+echo "=== (${PYTHON_VERSION}) Building Arrow C++ libraries ==="
+: ${ARROW_DATASET:=ON}
+: ${ARROW_FLIGHT:=ON}
+: ${ARROW_GANDIVA:=OFF}
+: ${ARROW_HDFS:=ON}
+: ${ARROW_JEMALLOC:=ON}
+: ${ARROW_MIMALLOC:=ON}
+: ${ARROW_ORC:=ON}
+: ${ARROW_PARQUET:=ON}
+: ${ARROW_PLASMA:=ON}
+: ${ARROW_S3:=ON}
+: ${ARROW_TENSORFLOW:=ON}
+: ${ARROW_WITH_BROTLI:=ON}
+: ${ARROW_WITH_BZ2:=ON}
+: ${ARROW_WITH_LZ4:=ON}
+: ${ARROW_WITH_SNAPPY:=ON}
+: ${ARROW_WITH_ZLIB:=ON}
+: ${ARROW_WITH_ZSTD:=ON}
+: ${CMAKE_BUILD_TYPE:=release}
+: ${CMAKE_GENERATOR:=Ninja}
+
+mkdir /tmp/arrow-build
+pushd /tmp/arrow-build
+cmake \
+ -DARROW_BROTLI_USE_SHARED=OFF \
+ -DARROW_BUILD_SHARED=ON \
+ -DARROW_BUILD_STATIC=OFF \
+ -DARROW_BUILD_TESTS=OFF \
+ -DARROW_DATASET=${ARROW_DATASET} \
+ -DARROW_DEPENDENCY_SOURCE="SYSTEM" \
+ -DARROW_DEPENDENCY_USE_SHARED=OFF \
+ -DARROW_FLIGHT==${ARROW_FLIGHT} \
+ -DARROW_GANDIVA=${ARROW_GANDIVA} \
+ -DARROW_HDFS=${ARROW_HDFS} \
+ -DARROW_JEMALLOC=${ARROW_JEMALLOC} \
+ -DARROW_MIMALLOC=${ARROW_MIMALLOC} \
+ -DARROW_ORC=${ARROW_ORC} \
+ -DARROW_PACKAGE_KIND="manylinux${MANYLINUX_VERSION}" \
+ -DARROW_PARQUET=${ARROW_PARQUET} \
+ -DARROW_PLASMA=${ARROW_PLASMA} \
+ -DARROW_PYTHON=ON \
+ -DARROW_RPATH_ORIGIN=ON \
+ -DARROW_S3=${ARROW_S3} \
+ -DARROW_TENSORFLOW=${ARROW_TENSORFLOW} \
+ -DARROW_UTF8PROC_USE_SHARED=OFF \
+ -DARROW_WITH_BROTLI=${ARROW_WITH_BROTLI} \
+ -DARROW_WITH_BZ2=${ARROW_WITH_BZ2} \
+ -DARROW_WITH_LZ4=${ARROW_WITH_LZ4} \
+ -DARROW_WITH_SNAPPY=${ARROW_WITH_SNAPPY} \
+ -DARROW_WITH_ZLIB=${ARROW_WITH_ZLIB} \
+ -DARROW_WITH_ZSTD=${ARROW_WITH_ZSTD} \
+ -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} \
+ -DCMAKE_INSTALL_LIBDIR=lib \
+ -DCMAKE_INSTALL_PREFIX=/tmp/arrow-dist \
+ -DCMAKE_TOOLCHAIN_FILE=/opt/vcpkg/scripts/buildsystems/vcpkg.cmake \
+ -DCMAKE_UNITY_BUILD=OFF \
Review comment:
Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553961230
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
We should keep as-is until someone visits with a strong *and* reasonable opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553926908
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
In the case of OpenSSL, if we bundled a shared version the version in the main system will be preferred. Not sure though where this is updated from. In `conda-forge` we explicitly have a Python setup that takes care that one is not used.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
Not sure about this. I don't know where the system one is coming from.
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
We should keep as-is until someone visits with a strong *and* reasonable opinion.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756989524
The OSX issue is that `protobuf` is installed through `brew` and that doesn't install it using `cmake`. You could conditionally add the following line https://github.com/apache/arrow/blob/a576cd71c08c3fec029d24092c407ece1775f5a6/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2459 only when protobuf is also build using bundled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-754252075
Related works:
* https://github.com/apache/arrow/pull/8386#issuecomment-728321361
* #8881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758677860
@github-actions crossbow -c ursacomputing/crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r554240959
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
Review comment:
How about prepend `python-` because `python-` exists in related names?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
Review comment:
How about prepend `python-` because `python-` exists in related names?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2014_x86_64?tab=tags
+ python-wheel-manylinux-2014:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ build:
+ args:
+ base: quay.io/pypa/manylinux2014_x86_64:2020-11-11-bc8ce45
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2014
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2014
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2014-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
+
+ python-wheel-manylinux-test:
+ image: python:${PYTHON}
+ shm_size: 2G
+ volumes:
+ - .:/arrow:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_test.sh
+
+ python-wheel-windows-vs2017:
+ image: ${REPO}:python-${PYTHON}-wheel-windows-vs2017
+ build:
+ args:
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-windows-vs2017.dockerfile
+ # This should make the pushed images reusable, but the image gets rebuilt
+ # no matter
+ # cache_from:
+ # - mcr.microsoft.com/windows/servercore:ltsc2019
+ # - ${REPO}:wheel-windows-vs2017
+ volumes:
+ - "${DOCKER_VOLUME_PREFIX}wheel-windows-clcache:C:/clcache"
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_build.bat
+
+ python-wheel-windows-test:
+ image: python:${PYTHON}-windowsservercore-1809
+ volumes:
+ - type: bind
+ source: .
+ target: "C:/arrow"
+ command: arrow\\ci\\scripts\\wheel_windows_test.bat
Review comment:
How about moving `python-wheel-*` services into `...### Python ###...` section?
##########
File path: docker-compose.yml
##########
@@ -120,9 +117,113 @@ x-hierarchy:
# helper services
- impala
- postgres
+ - python-wheel-manylinux-2010
+ - python-wheel-manylinux-2014
+ - python-wheel-manylinux-test
+ - python-wheel-windows-vs2017
+ - python-wheel-windows-test
+
+volumes:
+ conda-ccache:
+ name: ${ARCH}-conda-ccache
+ debian-ccache:
+ name: ${ARCH}-debian-${DEBIAN}-ccache
+ ubuntu-ccache:
+ name: ${ARCH}-ubuntu-${UBUNTU}-ccache
+ fedora-ccache:
+ name: ${ARCH}-fedora-${FEDORA}-ccache
+ debian-rust:
+ name: ${ARCH}-debian-${DEBIAN}-rust
+ maven-cache:
+ name: maven-cache
+ wheel-manylinux1-ccache:
+ name: centos-python-manylinux1-ccache
+ wheel-manylinux2010-ccache:
+ name: centos-python-manylinux2010-ccache
+ wheel-manylinux2014-ccache:
+ name: centos-python-manylinux2014-ccache
+ wheel-windows-clcache:
+ name: windows-wheel-clcache
services:
+ # See available versions at:
+ # https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ python-wheel-manylinux-2010:
+ image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ build:
+ args:
+ base: quay.io/pypa/manylinux2010_x86_64:2020-12-03-912b0de
+ vcpkg: ${VCPKG}
+ python: ${PYTHON}
+ context: .
+ dockerfile: ci/docker/wheel-manylinux-201x.dockerfile
+ cache_from:
+ - ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ environment:
+ <<: *ccache
+ MANYLINUX_VERSION: 2010
+ volumes:
+ - .:/arrow:delegated
+ - ${DOCKER_VOLUME_PREFIX}wheel-manylinux2010-ccache:/ccache:delegated
+ command: /arrow/ci/scripts/wheel_manylinux_build.sh
Review comment:
How about prepend `python_` because `python-` exists in related names?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553972503
##########
File path: python/setup.py
##########
@@ -385,17 +385,19 @@ def _bundle_arrow_cpp(self, build_prefix, build_lib):
build_prefix, build_lib,
"{}_regex".format(self.boost_namespace),
implib_required=False)
- if sys.platform == 'win32':
- # zlib uses zlib.dll for Windows
- zlib_lib_name = 'zlib'
- move_shared_libs(build_prefix, build_lib, zlib_lib_name,
- implib_required=False)
- if self.with_flight:
- # DLL dependencies for gRPC / Flight
- for lib_name in ['libcrypto-1_1-x64',
- 'libssl-1_1-x64']:
- move_shared_libs(build_prefix, build_lib, lib_name,
- implib_required=False)
+ # TODO(kszucs): consider to bundle the following shared libraries
Review comment:
Ok, I'm going to clean up the commented section used for bundling from `python/CMakeLists.txt` and `setup.py` then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#discussion_r553982246
##########
File path: ci/scripts/wheel_windows_build.bat
##########
@@ -0,0 +1,92 @@
+echo "Building windows wheel..."
+
+call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvars64.bat"
+
+echo "=== (%PYTHON_VERSION%) Clear output directories and leftovers ==="
+del /s /q C:\arrow-build
+del /s /q C:\arrow-dist
+del /s /q C:\arrow\python\dist
+del /s /q C:\arrow\python\build
+del /s /q C:\arrow\python\pyarrow\*.so
+del /s /q C:\arrow\python\pyarrow\*.so.*
+
+echo "=== (%PYTHON_VERSION%) Building Arrow C++ libraries ==="
+set ARROW_DATASET=ON
+set ARROW_FLIGHT=ON
+set ARROW_GANDIVA=OFF
+set ARROW_HDFS=ON
+set ARROW_ORC=OFF
+set ARROW_PARQUET=ON
+set ARROW_MIMALLOC=ON
+set ARROW_S3=ON
+set ARROW_TENSORFLOW=ON
+set ARROW_WITH_BROTLI=ON
+set ARROW_WITH_BZ2=ON
+set ARROW_WITH_LZ4=ON
+set ARROW_WITH_SNAPPY=ON
+set ARROW_WITH_ZLIB=ON
+set ARROW_WITH_ZSTD=ON
+set CMAKE_UNITY_BUILD=ON
+set CMAKE_GENERATOR=Visual Studio 15 2017 Win64
+
+mkdir C:\arrow-build
+pushd C:\arrow-build
+cmake ^
+ -DARROW_BUILD_SHARED=ON ^
+ -DARROW_BUILD_STATIC=OFF ^
+ -DARROW_BUILD_TESTS=OFF ^
+ -DARROW_CXXFLAGS="/MP" ^
+ -DARROW_DATASET=%ARROW_DATASET% ^
+ -DARROW_DEPENDENCY_SOURCE=SYSTEM ^
+ -DARROW_DEPENDENCY_USE_SHARED=OFF ^
+ -DARROW_FLIGHT=%ARROW_FLIGHT% ^
+ -DARROW_GANDIVA=%ARROW_GANDIVA% ^
+ -DARROW_HDFS=%ARROW_HDFS% ^
+ -DARROW_MIMALLOC=%ARROW_MIMALLOC% ^
+ -DARROW_ORC=%ARROW_ORC% ^
+ -DARROW_PACKAGE_KIND="wheel-windows" ^
+ -DARROW_PARQUET=%ARROW_PARQUET% ^
+ -DARROW_PYTHON=ON ^
+ -DARROW_S3=%ARROW_S3% ^
+ -DARROW_TENSORFLOW=%ARROW_TENSORFLOW% ^
+ -DARROW_WITH_BROTLI=%ARROW_WITH_BROTLI% ^
+ -DARROW_WITH_BZ2=%ARROW_WITH_BZ2% ^
+ -DARROW_WITH_LZ4=%ARROW_WITH_LZ4% ^
+ -DARROW_WITH_SNAPPY=%ARROW_WITH_SNAPPY% ^
+ -DARROW_WITH_ZLIB=%ARROW_WITH_ZLIB% ^
+ -DARROW_WITH_ZSTD=%ARROW_WITH_ZSTD% ^
+ -DCMAKE_BUILD_TYPE=%CMAKE_BUILD_TYPE% ^
+ -DLZ4_MSVC_LIB_PREFIX="" ^
+ -DLZ4_MSVC_STATIC_LIB_SUFFIX="" ^
+ -DZSTD_MSVC_LIB_PREFIX="" ^
+ -DCMAKE_CXX_COMPILER=clcache ^
+ -DCMAKE_INSTALL_PREFIX=C:\arrow-dist ^
+ -DCMAKE_TOOLCHAIN_FILE=C:\vcpkg\scripts\buildsystems\vcpkg.cmake ^
+ -DCMAKE_UNITY_BUILD=%CMAKE_UNITY_BUILD% ^
+ -DMSVC_LINK_VERBOSE=ON ^
+ -DVCPKG_TARGET_TRIPLET=x64-windows-static-md-%CMAKE_BUILD_TYPE% ^
+ -G "%CMAKE_GENERATOR%" ^
+ C:\arrow\cpp || exit /B
+cmake --build . --config %CMAKE_BUILD_TYPE% --target install || exit /B
+popd
+
+echo "=== (%PYTHON_VERSION%) Building wheel ==="
+set PYARROW_BUILD_TYPE=%CMAKE_BUILD_TYPE%
+set PYARROW_BUNDLE_ARROW_CPP=ON
+set PYARROW_BUNDLE_BOOST=OFF
+set PYARROW_CMAKE_GENERATOR=%CMAKE_GENERATOR%
+set PYARROW_INSTALL_TESTS=ON
+set PYARROW_WITH_DATASET=%ARROW_DATASET%
+set PYARROW_WITH_FLIGHT=%ARROW_FLIGHT%
+set PYARROW_WITH_GANDIVA=%ARROW_GANDIVA%
+set PYARROW_WITH_HDFS=%ARROW_HDFS%
+set PYARROW_WITH_ORC=%ARROW_ORC%
+set PYARROW_WITH_PARQUET=%ARROW_PARQUET%
+set PYARROW_WITH_S3=%ARROW_S3%
+set ARROW_HOME=C:\arrow-dist
+
+pushd C:\arrow\python
+@REM bundle the msvc runtime
+cp "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Redist\MSVC\14.16.27012\x64\Microsoft.VC141.CRT\msvcp140.dll" pyarrow\
Review comment:
Created a follow-up jira https://issues.apache.org/jira/browse/ARROW-11185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9096: ARROW-11212: [Packaging][Python] Use vcpkg as dependency source for manylinux and windows wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-758287910
https://issues.apache.org/jira/browse/ARROW-11212
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756449739
The build also executes the tests in a different docker container, see the "Test Wheel" step in the [build](https://github.com/ursa-labs/crossbow/runs/1664830035?check_suite_focus=true).
The following build should properly upload the windows wheel to crossbow releases now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9096: [Python][Packaging] Refactor manylinux and windows wheel building [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9096:
URL: https://github.com/apache/arrow/pull/9096#issuecomment-756738047
@github-actions crossbow submit wheel-manylinux2010-cp36m wheel-manylinux2014-cp36m wheel-windows-cp36m
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org