You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/21 19:46:26 UTC
[GitHub] [arrow] kszucs opened a new pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
kszucs opened a new pull request #9285:
URL: https://github.com/apache/arrow/pull/9285
This should work in theory, I'm building the image to see how well does vcpkg support arm packages.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-801838027
> I see that #8491 has been closed in favor of this PR.
>
> I'm far for being able to fully undertsand what's going on, but I would love a wheel for building `pyarrow` on the ARM AWS architecture.
>
> Do you think a future release of `pyarrow` will allow a simple `pip install pyarrow` to work on these rather new machines? Any idea when if that is so?
@Jaudouard Theoretically yes, right after the next major release. Until that you can try to install unofficial nightly wheels as described here https://arrow.apache.org/docs/python/install.html#installing-nightly-packages (note that you need a recent pip version for that).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592739245
##########
File path: dev/tasks/linux-packages/github.linux.amd64.yml
##########
@@ -73,7 +73,6 @@ jobs:
sudo apt update
sudo apt install -y \
apt-utils \
- createrepo \
Review comment:
It's used to test RPM packages.
Could you revert this?
##########
File path: dev/tasks/linux-packages/travis.linux.arm64.yml
##########
@@ -15,16 +15,17 @@
# specific language governing permissions and limitations
# under the License.
+arch: arm64-graviton2
+virt: vm
os: linux
-arch: arm64
-dist: bionic
+dist: focal
+group: edge
language: minimal
addons:
apt:
packages:
- apt-utils
- - createrepo
Review comment:
We need `createrepo` or `createrepo_c` for testing RPM packages.
`createrepo` exists only on Ubuntu 16.04 and 18.04.
`createrepo_c` exists only on Ubuntu 20.10 or later.
https://packages.ubuntu.com/search?keywords=createrepo
Can we use `bionic` with `arm64-graviton2`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593145300
##########
File path: dev/tasks/linux-packages/github.linux.amd64.yml
##########
@@ -73,7 +73,6 @@ jobs:
sudo apt update
sudo apt install -y \
apt-utils \
- createrepo \
Review comment:
Resolved
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593145497
##########
File path: ci/docker/python-wheel-manylinux-201x.dockerfile
##########
@@ -62,12 +65,14 @@ ARG build_type=release
ENV CMAKE_BUILD_TYPE=${build_type} \
VCPKG_FORCE_SYSTEM_BINARIES=1 \
VCPKG_OVERLAY_TRIPLETS=/arrow/ci/vcpkg \
- VCPKG_DEFAULT_TRIPLET=x64-linux-static-${build_type} \
+ VCPKG_DEFAULT_TRIPLET=${arch_short_alias}-linux-static-${build_type} \
VCPKG_FEATURE_FLAGS=-manifests
# TODO(kszucs): factor out the package enumeration to a text file and reuse it
# from the windows image and potentially in a future macos wheel build
RUN vcpkg install --clean-after-build \
+ boost-build:${arch_short_alias}-linux && \
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797221235
Revision: e3fe91e0c26452293b706d7b594165146ff8ca0b
Submitted crossbow builds: [ursacomputing/crossbow @ actions-216](https://github.com/ursacomputing/crossbow/branches/all?query=actions-216)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Jaudouard commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Jaudouard commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-801844577
Thanks for your anwer and for the good work @kszucs ! I will keep an eye on the releases then, and perhaps test a nightly build in the meantime.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592726508
##########
File path: dev/tasks/python-wheels/travis.linux.arm64.yml
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+arch: arm64-graviton2
+virt: vm
+os: linux
+dist: focal
+group: edge
+language: minimal
+
+addons:
+ apt:
+ packages:
+ - libgit2-dev
+ - python3-pip
+
+services:
+ - docker
+
+# don't build twice
+if: tag IS blank
+
+env:
+ global:
+ - BUILD_REF={{ arrow.head }}
+ - TRAVIS_TAG={{ task.tag }}
+ # archery uses these environment variables
+ - ARCH=arm64v8
+ - PYTHON={{ python_version }}
+
+before_script:
+ - git clone --no-checkout {{ arrow.remote }} arrow
+ - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ - git -C arrow checkout FETCH_HEAD
+ - git -C arrow submodule update --init --recursive
+
+script:
+ # Install Archery and Crossbow dependencies
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install docker-compose
+ - sudo -H pip3 install -e arrow/dev/archery[docker]
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
+
+ # Docker Login:
+ - echo "${DOCKERHUB_TOKEN}" | docker login --username "${DOCKERHUB_USER}" --password-stdin
+
+ # Build and Test packages
+
+ # output something every minutes to prevent travis from killing the build
+ - while sleep 1m; do echo "=====[ $SECONDS seconds still running ]====="; done &
+
+ - archery docker run -e SETUPTOOLS_SCM_PRETEND_VERSION={{ arrow.no_rc_version }} python-wheel-manylinux-{{ manylinux_version }}
+ - archery docker run python-wheel-manylinux-test-imports
+ - archery docker run python-wheel-manylinux-test-unittests
+
+ - archery docker push python-wheel-manylinux-{{ manylinux_version }}
+ - archery docker push python-wheel-manylinux-test-unittests
+
+ - kill %1
+
+after_success:
+ # Upload wheel as github artifact
+ - |
+ python3 arrow/dev/tasks/crossbow.py \
+ --queue-path . \
+ --queue-remote {{ queue_remote_url }} \
+ upload-artifacts \
+ --pattern "arrow/python/repaired_wheels/*.whl"
+ --sha {{ task.branch }} \
+ --tag {{ task.tag }}
+
+ # Push the docker image
+ - archery docker push python-wheel-manylinux-{{ manylinux_version }}
Review comment:
Remove this before merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it. Is that an option? if so, I'll document the steps.
I'm proceeding with buillding manually, seeing what the next issues are. Currently turned Gandiva component OFF (something with llvm for aarch64) and now seeing some issue with Plasma.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797232216
Revision: 56449abb805035388c7f1213fab36d97477bd419
Submitted crossbow builds: [ursacomputing/crossbow @ actions-217](https://github.com/ursacomputing/crossbow/branches/all?query=actions-217)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797546572
Revision: 599f9fbfde615553c08eebb31e9cb2d4923192a1
Submitted crossbow builds: [ursacomputing/crossbow @ actions-223](https://github.com/ursacomputing/crossbow/branches/all?query=actions-223)
|Task|Status|
|----|------|
|wheel-manylinux2014-cp36m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797074051
@github-actions crossbow submit -g linux-arm64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592746024
##########
File path: dev/tasks/python-wheels/github.linux.amd64.yml
##########
@@ -95,7 +87,18 @@ jobs:
env:
CROSSBOW_GITHUB_TOKEN: {{ '${{ secrets.CROSSBOW_GITHUB_TOKEN }}' }}
+ # TODO(kszucs): only push from master branch
+ - name: Push the docker image
+ shell: bash
+ run: archery docker push python-wheel-manylinux-{{ manylinux_version }}
Review comment:
Can we remove this step before we merge this?
##########
File path: dev/tasks/python-wheels/travis.linux.arm64.yml
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+arch: arm64-graviton2
+virt: vm
+os: linux
+dist: focal
+group: edge
+language: minimal
+
+addons:
+ apt:
+ packages:
+ - libgit2-dev
+ - python3-pip
+
+services:
+ - docker
+
+# don't build twice
+if: tag IS blank
+
+env:
+ global:
+ - BUILD_REF={{ arrow.head }}
+ - TRAVIS_TAG={{ task.tag }}
+ # archery uses these environment variables
+ - ARCH=arm64v8
+ - PYTHON={{ python_version }}
+
+before_script:
+ - git clone --no-checkout {{ arrow.remote }} arrow
+ - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ - git -C arrow checkout FETCH_HEAD
+ - git -C arrow submodule update --init --recursive
+
+script:
+ # Install Archery and Crossbow dependencies
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install docker-compose
+ - sudo -H pip3 install -e arrow/dev/archery[docker]
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
+
+ # Docker Login:
+ - echo "${DOCKERHUB_TOKEN}" | docker login --username "${DOCKERHUB_USER}" --password-stdin
+
+ # Build and Test packages
+
+ # output something every minutes to prevent travis from killing the build
+ - while sleep 1m; do echo "=====[ $SECONDS seconds still running ]====="; done &
Review comment:
Wow!!!
##########
File path: ci/docker/python-wheel-manylinux-201x.dockerfile
##########
@@ -62,12 +65,14 @@ ARG build_type=release
ENV CMAKE_BUILD_TYPE=${build_type} \
VCPKG_FORCE_SYSTEM_BINARIES=1 \
VCPKG_OVERLAY_TRIPLETS=/arrow/ci/vcpkg \
- VCPKG_DEFAULT_TRIPLET=x64-linux-static-${build_type} \
+ VCPKG_DEFAULT_TRIPLET=${arch_short_alias}-linux-static-${build_type} \
VCPKG_FEATURE_FLAGS=-manifests
# TODO(kszucs): factor out the package enumeration to a text file and reuse it
# from the windows image and potentially in a future macos wheel build
RUN vcpkg install --clean-after-build \
+ boost-build:${arch_short_alias}-linux && \
Review comment:
Can you keep this restliction as a comment in this file?
##########
File path: dev/tasks/tasks.yml
##########
@@ -121,9 +121,9 @@ groups:
######################## Tasks to run regularly #############################
nightly:
- - debian-*-amd64
- - ubuntu-*-amd64
- - centos-*-amd64
+ - debian-*
Review comment:
Is it OK that Ursa Computing keeps Travis CI subscription?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796871774
@github-actions crossbow submit wheel-manylinux-* debian-*-arm64 ubuntu-*-arm64 centos-*-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy @kszucs hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it - I cloned latest 1.8.129 tag rather than 1.8.126 which vcpkg currently pulls). Is that an option? if so, I'll document the steps.
I've proceeded with buillding manually, to see what the next issues are:
* Turned `-DARROW_GANDIVA=OFF` (some issue with llvm for aarch64)
* Did not attempt to turn on CUDA yet
Following the above -> Arrow C++ libs were built. Tests passed except for:
```
The following tests FAILED:
41 - arrow-s3fs-test (Failed)
```
(didn't go into that yet)
Moving on to PyArrow, built with all components except `PYARROW_WITH_GANDIVA` and `CUDA`:
* Building with `PYARROW_WITH_TENSORFLOW` - I got [this existing issue](https://github.com/apache/arrow/issues/7065) when running tests
* Turning off TF component, build passed.
* Tests passed, some are skipped:
`3329 passed, 351 skipped, 17 xfailed, 49 warnings in 53.40s`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592725973
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
@xhochy despite that the build stage uses the same numpy versions, I'm getting `the` binary incompatibility error: `numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject`, but just on py39.
You may know the root cause of it, see the build log at https://travis-ci.com/github/ursacomputing/crossbow/builds/219762923#L3832
With 1.20 it works just fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797151579
Revision: 28e75dcc452522f21a04263f10fa1db63b8fea00
Submitted crossbow builds: [ursacomputing/crossbow @ actions-214](https://github.com/ursacomputing/crossbow/branches/all?query=actions-214)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797077627
The arm machines should be available for open source repositories https://docs.travis-ci.com/user/multi-cpu-architectures/#multi-cpu-availaibility
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593112806
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
cc @pitrou
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797055897
@kou I also resolved travis related problems* with the linux package builds, though the centos ones are failing for different reasons. Could you take a look at those?
*stdout buffering makes the build halt unless we use the graviton2 machines, if we have `arch: arm64-graviton2` in the config but don't add `group: edge` as well then travis will spin up amd64 instances
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593170288
##########
File path: .env
##########
@@ -33,9 +33,13 @@ COMPOSE_DOCKER_CLI_BUILD=1
DOCKER_BUILDKIT=1
BUILDKIT_INLINE_CACHE=1
+# different architecture notations
+ARCH=amd64
+ARCH_ALIAS=x86_64
+ARCH_SHORT_ALIAS=x64
Review comment:
That's done by archery, but placing the defaults in `.env` enables the usage of docker-compose without archery.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593147612
##########
File path: .env
##########
@@ -33,9 +33,13 @@ COMPOSE_DOCKER_CLI_BUILD=1
DOCKER_BUILDKIT=1
BUILDKIT_INLINE_CACHE=1
+# different architecture notations
+ARCH=amd64
+ARCH_ALIAS=x86_64
+ARCH_SHORT_ALIAS=x64
Review comment:
Can the aliases be deduced automatically from ARCH? It would be more user-friendly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797158422
Revision: 965e6964366a832042564d7ccb6a0d44849f233a
Submitted crossbow builds: [ursacomputing/crossbow @ actions-215](https://github.com/ursacomputing/crossbow/branches/all?query=actions-215)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592727203
##########
File path: dev/tasks/python-wheels/github.linux.amd64.yml
##########
@@ -29,18 +29,9 @@ jobs:
name: "Build wheel for Manylinux {{ manylinux_version }}"
runs-on: ubuntu-latest
env:
- # archery uses this environment variable
+ # archery uses these environment variables
+ ARCH: amd64
PYTHON: {{ python_version }}
- # this is a private repository at the moment (mostly because of licensing
- # consideration of windows images with visual studio), but anyone can
- # recreate the image by manually building it via:
- # `archery build python-wheel-manylinux-2010|2014`
- # note that we don't run docker build since there wouldn't be a cache hit
- # and rebuilding the dependencies takes a fair amount of time
- # (in case of linux multi-host caching works much better with buildkit)
- REPO: ghcr.io/ursacomputing/arrow
Review comment:
Changed it to use the more widely accessible apache dockerhub repo (originally changed to use a private repo because of the windows images).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs closed pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs closed pull request #9285:
URL: https://github.com/apache/arrow/pull/9285
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796880889
Revision: 08487baa14e6214240184d0c16ee58e18f02941f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-211](https://github.com/ursacomputing/crossbow/branches/all?query=actions-211)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2010-cp36m-amd64)|
|wheel-manylinux2010-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2010-cp37m-amd64)|
|wheel-manylinux2010-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2010-cp38-amd64)|
|wheel-manylinux2010-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2010-cp39-amd64)|
|wheel-manylinux2014-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2014-cp36m-amd64)|
|wheel-manylinux2014-cp36m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2014-cp37m-amd64)|
|wheel-manylinux2014-cp37m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2014-cp38-amd64)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-211-github-wheel-manylinux2014-cp39-amd64)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-764893560
https://issues.apache.org/jira/browse/ARROW-10349
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592720028
##########
File path: dev/tasks/tasks.yml
##########
@@ -121,9 +121,9 @@ groups:
######################## Tasks to run regularly #############################
nightly:
- - debian-*-amd64
- - ubuntu-*-amd64
- - centos-*-amd64
+ - debian-*
Review comment:
I managed to make the arm builds more stable on travis using graviton machines, so I think we should exercise these on a nightly basis.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797297334
Revision: 68833b79ee9911ebfc5fd3e986b34c2d7d1f1f04
Submitted crossbow builds: [ursacomputing/crossbow @ actions-219](https://github.com/ursacomputing/crossbow/branches/all?query=actions-219)
|Task|Status|
|----|------|
|centos-8-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-814674585
@Yikun Can you add `--prefer-binary` to your `pip` command line?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797250627
@github-actions crossbow submit centos-7-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593148810
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
It may be a Pandas build problem @jorisvandenbossche
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797250897
Revision: 68833b79ee9911ebfc5fd3e986b34c2d7d1f1f04
Submitted crossbow builds: [ursacomputing/crossbow @ actions-218](https://github.com/ursacomputing/crossbow/branches/all?query=actions-218)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797296991
@github-actions crossbow submit centos-8-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797085660
Revision: d1ef8799a4c9a98e6245e5329e56b28effb193d2
Submitted crossbow builds: [ursacomputing/crossbow @ actions-213](https://github.com/ursacomputing/crossbow/branches/all?query=actions-213)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|centos-8-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|debian-buster-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-bionic-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-focal-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-groovy-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797055897
@kou I also resolved travis related problems with the linux package builds, though the centos ones are failing for different reasons. Could you take a look at those?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Yikun commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-814680631
@pitrou Thanks for you super fast reply, and it works. the complete cmd like:
```
pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --pre pyarrow --prefer-binary
```
Do you know any official release plan of pyarrow?
Note that, the pyspark [is using pyarrow](https://github.com/apache/spark/blob/0aa2c284e4052cb57ebf7276ecc4867ea2d5f94f/python/setup.py#L259), but its installation would be breaked by pyarrow in aarch64 env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592728500
##########
File path: ci/docker/python-wheel-manylinux-201x.dockerfile
##########
@@ -62,12 +65,14 @@ ARG build_type=release
ENV CMAKE_BUILD_TYPE=${build_type} \
VCPKG_FORCE_SYSTEM_BINARIES=1 \
VCPKG_OVERLAY_TRIPLETS=/arrow/ci/vcpkg \
- VCPKG_DEFAULT_TRIPLET=x64-linux-static-${build_type} \
+ VCPKG_DEFAULT_TRIPLET=${arch_short_alias}-linux-static-${build_type} \
VCPKG_FEATURE_FLAGS=-manifests
# TODO(kszucs): factor out the package enumeration to a text file and reuse it
# from the windows image and potentially in a future macos wheel build
RUN vcpkg install --clean-after-build \
+ boost-build:${arch_short_alias}-linux && \
Review comment:
Need to install the boost-build prior installing the boost packages, otherwise vcpkg will raise an error.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592722741
##########
File path: docker-compose.yml
##########
@@ -122,7 +122,8 @@ x-hierarchy:
- postgres
- python-wheel-manylinux-2010
- python-wheel-manylinux-2014
- - python-wheel-manylinux-test
+ - python-wheel-manylinux-test-imports
Review comment:
I had to separate the two testing steps due to the lack of pandas wheels for aarch64. Compiling pandas takes significant time, so I rather chose to cache it in a docker image.
Installing the pyarrow wheels in an environment with preinstalled dependencies like numpy wouldn't test the setup.py config, so IMO it's better to isolate these to two separate cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797317650
Revision: 68833b79ee9911ebfc5fd3e986b34c2d7d1f1f04
Submitted crossbow builds: [ursacomputing/crossbow @ actions-220](https://github.com/ursacomputing/crossbow/branches/all?query=actions-220)
|Task|Status|
|----|------|
|centos-7-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-220-github-centos-7-amd64)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797209122
@github-actions crossbow submit centos-7-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796939954
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
eladroz commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-799367309
eager to test this, thx for the work @kszucs! π
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592727552
##########
File path: dev/tasks/linux-packages/travis.linux.arm64.yml
##########
@@ -104,7 +105,8 @@ script:
- popd
after_success:
- - sudo -H pip3 install --requirement arrow/dev/tasks/requirements-crossbow.txt
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
Review comment:
We need specific pygit2 versions according to the system's libgit2 version.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797231938
@github-actions crossbow submit centos-7-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592725973
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
@xhochy despite that the build stage uses the same numpy versions, I'm getting `the` binary incompatibility error: `numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject`, but just on py39.
You may know the root cause of it, see the build log at https://travis-ci.com/github/ursacomputing/crossbow/builds/219762923#L3832
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-765333956
@kszucs What is the problem with AWS, can you post the linker error here? Otherwise this is quite neat π
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797448591
Revision: 4e56f9d977a75a1f4d6d30f5a9defce990d37743
Submitted crossbow builds: [ursacomputing/crossbow @ actions-222](https://github.com/ursacomputing/crossbow/branches/all?query=actions-222)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2010-cp36m-amd64)|
|wheel-manylinux2010-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2010-cp37m-amd64)|
|wheel-manylinux2010-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2010-cp38-amd64)|
|wheel-manylinux2010-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2010-cp39-amd64)|
|wheel-manylinux2014-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2014-cp36m-amd64)|
|wheel-manylinux2014-cp36m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2014-cp37m-amd64)|
|wheel-manylinux2014-cp37m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2014-cp38-amd64)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-222-github-wheel-manylinux2014-cp39-amd64)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592719057
##########
File path: dev/tasks/linux-packages/github.linux.amd64.yml
##########
@@ -73,7 +73,6 @@ jobs:
sudo apt update
sudo apt install -y \
apt-utils \
- createrepo \
Review comment:
@kou apt failed to locate createrepo on the graviton machines despite the same ubuntu
looked for its usage and seems like we're using it only from the binary release scripts, so I removed it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797153570
@github-actions crossbow submit centos-7-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797148017
It seems that `arm64-graviton2` doesn't provide `bionic`.
We need to build createrepo_c from source: https://github.com/rpm-software-management/createrepo_c
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-765362746
@xhochy a set of declaration issues with the static build:
```
In file included from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.inl:29, from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.h:202,
from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/array_list.h:9, from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/event-stream/event_stream.h:9, from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventHeader.h:14, from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/source/utils/event/EventHeader.cpp:6:
/opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.gcc_arm64_asm.inl:23:26: error: βuint64_t aws_mul_u64_saturating(uint64_t, uint64_t)β was declared βexternβ and later βstaticβ [-fpermissive] 23 | AWS_STATIC_IMPL uint64_t aws_mul_u64_saturating(uint64_t a, uint64_t b) { | ^~~~~~~~~~~~~~~~~~~~~~ In file included from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/array_list.h:9,
from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/event-stream/event_stream.h:9,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventHeader.h:14,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/source/utils/event/EventHeader.cpp:6:
/opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.h:46:30: note: previous declaration of βuint64_t aws_mul_u64_saturating(uint64_t, uint64_t)β
46 | AWS_COMMON_MATH_API uint64_t aws_mul_u64_saturating(uint64_t a, uint64_t b);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592745773
##########
File path: dev/tasks/linux-packages/travis.linux.arm64.yml
##########
@@ -15,16 +15,17 @@
# specific language governing permissions and limitations
# under the License.
+arch: arm64-graviton2
+virt: vm
os: linux
-arch: arm64
-dist: bionic
+dist: focal
+group: edge
language: minimal
addons:
apt:
packages:
- apt-utils
- - createrepo
Review comment:
Tried with bionic as well, but apt was unable to locate `createrepo`. Shall I try with `createrepo_c` on focal?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592727914
##########
File path: ci/scripts/python_wheel_manylinux_build.sh
##########
@@ -135,8 +137,5 @@ pushd /arrow/python
python setup.py bdist_wheel
echo "=== (${PYTHON_VERSION}) Tag the wheel with manylinux${MANYLINUX_VERSION} ==="
-auditwheel repair \
- --plat "manylinux${MANYLINUX_VERSION}_x86_64" \
Review comment:
As it turns out, the platform tag is detected automatically.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Jaudouard edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Jaudouard edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-801797963
I see that #8491 has been closed in favor of this PR.
I'm far for being able to fully undertsand what's going on, but I would love a wheel for building `pyarrow` on the ARM AWS architecture.
Do you think a future release of `pyarrow` will allow a simple `pip install pyarrow` to work on these rather new machines? Any idea when if that is so?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-814686747
We're planning a 4.0 release in a couple of weeks, no definite date.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy @kszucs hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it. Is that an option? if so, I'll document the steps.
I've proceeded with buillding manually, to see what the next issues are:
* Turned `-DARROW_GANDIVA=OFF` (some issue with llvm for aarch64)
* Needed to copy OpenSSL (`libcrypto.a`) that was fetched by vcpkg to my `/usr/local/lib64`, instead of the one built when compiling the AWS SDK locally (which seems to have an issue with arm-related symbol `OPENSSL_armcap_P`)
* Did not attempt to turn on CUDA yet
Following the above -> Arrow C++ libs were built. Tests passed except for:
```
The following tests FAILED:
41 - arrow-s3fs-test (Failed)
43 - arrow-flight-test (Failed)
50 - arrow-python-test (Failed)
```
(didn't go into that yet)
Moving on to PyArrow, built with all components except `PYARROW_WITH_GANDIVA` and `_CUDA`:
* Building with `PYARROW_WITH_TENSORFLOW` - I got [this existing issue](https://github.com/apache/arrow/issues/7065) when running tests
* Turning off TF component, build passed.
* Tests passed, some are skipped:
`3329 passed, 351 skipped, 17 xfailed, 49 warnings in 53.40s`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796879227
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797150568
@github-actions crossbow submit centos-7-aarch64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796878664
Revision: 08487baa14e6214240184d0c16ee58e18f02941f
Submitted crossbow builds: [ursacomputing/crossbow @ actions-210](https://github.com/ursacomputing/crossbow/branches/all?query=actions-210)
|Task|Status|
|----|------|
|centos-7-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|centos-8-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|debian-buster-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-bionic-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-focal-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-groovy-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592723936
##########
File path: python/requirements-wheel-build.txt
##########
@@ -1,5 +1,6 @@
cython>=0.29.11
-numpy==1.16.6; python_version < "3.9"
-numpy==1.19.4; python_version >= "3.9"
+numpy==1.16.6; python_version < "3.9" and platform_machine != "aarch64"
+numpy==1.19.4; python_version >= "3.9" or platform_machine == "aarch64"
Review comment:
There is no wheel available on aarch64 for earlier versions. We could compile numpy from source, but I'm not sure whether it worth it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-765362746
@xhochy a set of declaration issues with the static build:
```
/opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.gcc_arm64_asm.inl:166:26: error: βuint32_t aws_add_u32_saturating(uint32_t, uint32_t)β was declared βexternβ and later βstaticβ [-fpermissive]
166 | AWS_STATIC_IMPL uint32_t aws_add_u32_saturating(uint32_t a, uint32_t b) {
| ^~~~~~~~~~~~~~~~~~~~~~
In file included from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/array_list.h:9,
from /opt/vcpkg/installed/arm64-linux-static-release/include/aws/event-stream/event_stream.h:9,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventHeader.h:14,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventStreamHandler.h:10,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventStreamDecoder.h:10,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventStreamBuf.h:9,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventDecoderStream.h:8,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-core/include/aws/core/utils/event/EventStream.h:7,
from /opt/vcpkg/buildtrees/aws-sdk-cpp/src/1c85d0a8c9-fb0dd46991.clean/aws-cpp-sdk-s3/source/S3Client.cpp:20,
from aws-cpp-sdk-s3/ub_S3.cpp:3:
/opt/vcpkg/installed/arm64-linux-static-release/include/aws/common/math.h:79:30: note: previous declaration of βuint32_t aws_add_u32_saturating(uint32_t, uint32_t)β
79 | AWS_COMMON_MATH_API uint32_t aws_add_u32_saturating(uint32_t a, uint32_t b);
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593147811
##########
File path: .env
##########
@@ -33,9 +33,13 @@ COMPOSE_DOCKER_CLI_BUILD=1
DOCKER_BUILDKIT=1
BUILDKIT_INLINE_CACHE=1
+# different architecture notations
+ARCH=amd64
+ARCH_ALIAS=x86_64
+ARCH_SHORT_ALIAS=x64
Review comment:
Ha, nevermind, I see you did this below :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593112188
##########
File path: dev/tasks/python-wheels/github.linux.amd64.yml
##########
@@ -95,7 +87,18 @@ jobs:
env:
CROSSBOW_GITHUB_TOKEN: {{ '${{ secrets.CROSSBOW_GITHUB_TOKEN }}' }}
+ # TODO(kszucs): only push from master branch
+ - name: Push the docker image
+ shell: bash
+ run: archery docker push python-wheel-manylinux-{{ manylinux_version }}
Review comment:
Removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592733738
##########
File path: dev/tasks/python-wheels/travis.linux.arm64.yml
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+arch: arm64-graviton2
+virt: vm
+os: linux
+dist: focal
+group: edge
+language: minimal
+
+addons:
+ apt:
+ packages:
+ - libgit2-dev
+ - python3-pip
+
+services:
+ - docker
+
+# don't build twice
+if: tag IS blank
+
+env:
+ global:
+ - BUILD_REF={{ arrow.head }}
+ - TRAVIS_TAG={{ task.tag }}
+ # archery uses these environment variables
+ - ARCH=arm64v8
+ - PYTHON={{ python_version }}
+
+before_script:
+ - git clone --no-checkout {{ arrow.remote }} arrow
+ - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ - git -C arrow checkout FETCH_HEAD
+ - git -C arrow submodule update --init --recursive
+
+script:
+ # Install Archery and Crossbow dependencies
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install docker-compose
+ - sudo -H pip3 install -e arrow/dev/archery[docker]
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
+
+ # Docker Login:
+ - echo "${DOCKERHUB_TOKEN}" | docker login --username "${DOCKERHUB_USER}" --password-stdin
+
+ # Build and Test packages
+
+ # output something every minutes to prevent travis from killing the build
Review comment:
`travis_wait` doesn't work in case of ARM instances
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-792364856
@eladroz thanks for working on this. We should try to update our pinned vcpkg version and report any compilation issues upstream.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-764893560
https://issues.apache.org/jira/browse/ARROW-10349
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy @kszucs hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it - I cloned latest 1.7.x tag rather than 1.8.126 which vcpkg currently pulls). Is that an option? if so, I'll document the steps.
I've proceeded with buillding manually, to see what the next issues are:
* Turned `-DARROW_GANDIVA=OFF` (some issue with llvm for aarch64)
* Needed to copy OpenSSL (`libcrypto.a`) that was fetched by vcpkg to my `/usr/local/lib64`, instead of the one built when compiling the AWS SDK locally (which seems to have an issue with arm-related symbol `OPENSSL_armcap_P`)
* Did not attempt to turn on CUDA yet
Following the above -> Arrow C++ libs were built. Tests passed except for:
```
The following tests FAILED:
41 - arrow-s3fs-test (Failed)
43 - arrow-flight-test (Failed)
50 - arrow-python-test (Failed)
```
(didn't go into that yet)
Moving on to PyArrow, built with all components except `PYARROW_WITH_GANDIVA` and `_CUDA`:
* Building with `PYARROW_WITH_TENSORFLOW` - I got [this existing issue](https://github.com/apache/arrow/issues/7065) when running tests
* Turning off TF component, build passed.
* Tests passed, some are skipped:
`3329 passed, 351 skipped, 17 xfailed, 49 warnings in 53.40s`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy @kszucs hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it. Is that an option? if so, I'll document the steps.
I'm proceeding with buillding manually, seeing what the next issues are. Currently turned Gandiva component OFF (something with llvm for aarch64) and now seeing some issue with Plasma.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797438464
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797377305
Revision: fca5a0f2b6e4c4b116fea44369ba71afe866acf2
Submitted crossbow builds: [ursacomputing/crossbow @ actions-221](https://github.com/ursacomputing/crossbow/branches/all?query=actions-221)
|Task|Status|
|----|------|
|centos-7-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-centos-7-amd64)|
|centos-8-aarch64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|centos-8-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-centos-8-amd64)|
|debian-buster-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-debian-buster-amd64)|
|debian-buster-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-bionic-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-ubuntu-bionic-amd64)|
|ubuntu-bionic-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-focal-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-ubuntu-focal-amd64)|
|ubuntu-focal-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|ubuntu-groovy-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-221-github-ubuntu-groovy-amd64)|
|ubuntu-groovy-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593422984
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
Although numpy 1.20 should be the latest numpy anyway, at the moment. So that shouldn't matter?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592719583
##########
File path: dev/tasks/python-wheels/travis.linux.arm64.yml
##########
@@ -0,0 +1,99 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+arch: arm64-graviton2
+virt: vm
+os: linux
+dist: focal
+group: edge
+language: minimal
+
+addons:
+ apt:
+ packages:
+ - libgit2-dev
+ - python3-pip
+
+services:
+ - docker
+
+# don't build twice
+if: tag IS blank
+
+env:
+ global:
+ - BUILD_REF={{ arrow.head }}
+ - TRAVIS_TAG={{ task.tag }}
+ # archery uses these environment variables
+ - ARCH=arm64v8
+ - PYTHON={{ python_version }}
+
+before_script:
+ - git clone --no-checkout {{ arrow.remote }} arrow
+ - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ - git -C arrow checkout FETCH_HEAD
+ - git -C arrow submodule update --init --recursive
+
+script:
+ # Install Archery and Crossbow dependencies
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install docker-compose
+ - sudo -H pip3 install -e arrow/dev/archery[docker]
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
+
+ # Docker Login:
+ - echo "${DOCKERHUB_TOKEN}" | docker login --username "${DOCKERHUB_USER}" --password-stdin
+
+ # Build and Test packages
+
+ # output something every minutes to prevent travis from killing the build
+ - while sleep 1m; do echo "=====[ $SECONDS seconds still running ]====="; done &
+
+ - archery docker run -e SETUPTOOLS_SCM_PRETEND_VERSION={{ arrow.no_rc_version }} python-wheel-manylinux-{{ manylinux_version }}
+ - archery docker run python-wheel-manylinux-test-imports
+ - archery docker run python-wheel-manylinux-test-unittests
+
+ - archery docker push python-wheel-manylinux-{{ manylinux_version }}
Review comment:
Need to move the push commands to the "on master" blocks below before merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Yikun commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-814697042
@pitrou thanks for you info, I'm looking forward to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593421714
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
Yeah, pandas is building from source on aarch64, since we don't have a wheel for the latest version. And numpy isn't pinned in our pyproject.toml for python 3.9, I see now (which should get fixed)
##########
File path: python/requirements-wheel-test.txt
##########
@@ -1,10 +1,11 @@
cffi
cython
hypothesis
-numpy==1.19.4
-pandas<1.1.0; python_version < "3.8"
-pandas; python_version >= "3.8"
-pickle5; (python_version == "3.6" or python_version == "3.7") and sys_platform != 'win32'
+numpy==1.19.4; python_version != "3.9" or platform_machine != "aarch64"
+numpy==1.20.1; python_version == "3.9" and platform_machine == "aarch64"
Review comment:
Yeah, pandas is building from source on aarch64, since we don't have a wheel for the latest version. And numpy isn't pinned in our pyproject.toml for python 3.9, I see now (which should get fixed): https://github.com/pandas-dev/pandas/blob/59fe44e961c0496b3854b0edc6b11a887d30c13f/pyproject.toml#L12
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797317090
@github-actions crossbow submit centos-7-amd64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r592723165
##########
File path: docker-compose.yml
##########
@@ -695,17 +696,20 @@ services:
# See available versions at:
# https://quay.io/repository/pypa/manylinux2010_x86_64?tab=tags
+ # only amd64 arch is supported
python-wheel-manylinux-2010:
- image: ${REPO}:python-${PYTHON}-wheel-manylinux-2010
+ image: ${REPO}:${ARCH}-python-${PYTHON}-wheel-manylinux-2010
Review comment:
manylinux2010 doesn't actually support aarch64, but we reuse the same dockerfile for both manylinux specs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797347475
@github-actions crossbow submit -g linux
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-796990354
Revision: 29cbead4b589a5c57ff457c18e17656492a05397
Submitted crossbow builds: [ursacomputing/crossbow @ actions-212](https://github.com/ursacomputing/crossbow/branches/all?query=actions-212)
|Task|Status|
|----|------|
|wheel-manylinux2010-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2010-cp36m-amd64)|
|wheel-manylinux2010-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2010-cp37m-amd64)|
|wheel-manylinux2010-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2010-cp38-amd64)|
|wheel-manylinux2010-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2010-cp39-amd64)|
|wheel-manylinux2014-cp36m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2014-cp36m-amd64)|
|wheel-manylinux2014-cp36m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp37m-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2014-cp37m-amd64)|
|wheel-manylinux2014-cp37m-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp38-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2014-cp38-amd64)|
|wheel-manylinux2014-cp38-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
|wheel-manylinux2014-cp39-amd64|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-212-github-wheel-manylinux2014-cp39-amd64)|
|wheel-manylinux2014-cp39-arm64|[](https://travis-ci.org/ursacomputing/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Yikun commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Yikun commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-814672932
```shell
pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --pre pyarrow
```
The one line pyarrow nightly wheels installation is still not working in my aarch64 env.
The error output looks like:
```
Looking in indexes: https://pypi.org/simple, https://pypi.fury.io/arrow-nightlies/
Collecting pyarrow
Downloading https://pypi.fury.io/arrow-nightlies/-/ver_10iU12/pyarrow-3.1.0.dev538.tar.gz (706 kB)
|ββββββββββββββββββββββββββββββββ| 706 kB 213 kB/s
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Requirement already satisfied: numpy>=1.16.6 in /root/venv/lib/python3.8/site-packages (from pyarrow) (1.20.2)
Building wheels for collected packages: pyarrow
Building wheel for pyarrow (PEP 517) ... error
ERROR: Command errored out with exit status 1:
// ... ...
-- Could NOT find Arrow (missing: Arrow_DIR)
-- Checking for module 'arrow'
-- No package 'arrow' found
CMake Error at /usr/local/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:218 (message):
Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
Call Stack (most recent call first):
/usr/local/share/cmake-3.19/Modules/FindPackageHandleStandardArgs.cmake:582 (_FPHSA_FAILURE_MESSAGE)
cmake_modules/FindArrow.cmake:419 (find_package_handle_standard_args)
cmake_modules/FindArrowPython.cmake:46 (find_package)
CMakeLists.txt:218 (find_package)
-- Configuring incomplete, errors occurred!
See also "/tmp/pip-install-90jcajgr/pyarrow_6ee2e29916394a13ba2ac9eef03a9e44/build/temp.linux-aarch64-3.8/CMakeFiles/CMakeOutput.log".
error: command 'cmake' failed with exit status 1
----------------------------------------
ERROR: Failed building wheel for pyarrow
Failed to build pyarrow
ERROR: Could not build wheels for pyarrow which use PEP 517 and cannot be installed directly
```
Looks like I need to install the `arrow` manually? But I can use `pip install pyarrow` in my x86 env without any other separate installation.
Do you have any idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] Jaudouard commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
Jaudouard commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-801797963
I see that #8491 has been closed in favor of this PR.
I'm far for being able to fully undertsand what's going on, but I would love a wheel for building `pyarrow` on the ARM AWS architecture.
Do you think a future release of `pyarrow` will allow a simple pip install `pyarrow` to work on these rather new machines? Any idea when if that is so?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#discussion_r593444798
##########
File path: dev/tasks/python-wheels/travis.linux.arm64.yml
##########
@@ -0,0 +1,91 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+arch: arm64-graviton2
+virt: vm
+os: linux
+dist: focal
+group: edge
+language: minimal
+
+addons:
+ apt:
+ packages:
+ - libgit2-dev
+ - python3-pip
+
+services:
+ - docker
+
+# don't build twice
+if: tag IS blank
+
+env:
+ global:
+ - BUILD_REF={{ arrow.head }}
+ - TRAVIS_TAG={{ task.tag }}
+ # archery uses these environment variables
+ - ARCH=arm64v8
+ - PYTHON={{ python_version }}
+
+before_script:
+ - git clone --no-checkout {{ arrow.remote }} arrow
+ - git -C arrow fetch -t {{ arrow.remote }} {{ arrow.branch }}
+ - git -C arrow checkout FETCH_HEAD
+ - git -C arrow submodule update --init --recursive
+
+script:
+ # Install Archery and Crossbow dependencies
+ - sudo -H pip3 install --upgrade pip
+ - sudo -H pip3 install docker-compose
+ - sudo -H pip3 install -e arrow/dev/archery[docker]
+ - sudo -H pip3 install -r arrow/dev/tasks/requirements-crossbow.txt pygit2==1.0
+
+ # Docker Login:
+ - echo "${DOCKERHUB_TOKEN}" | docker login --username "${DOCKERHUB_USER}" --password-stdin
+
+ # Build and Test packages
+
+ # output something every minutes to prevent travis from killing the build
+ - while sleep 1m; do echo "=====[ $SECONDS seconds still running ]====="; done &
+ - archery docker run -e SETUPTOOLS_SCM_PRETEND_VERSION={{ arrow.no_rc_version }} python-wheel-manylinux-{{ manylinux_version }}
+ - archery docker run python-wheel-manylinux-test-imports
+ - archery docker run python-wheel-manylinux-test-unittests
+ - kill %1
+
+after_success:
+ # Upload wheel as github artifact
+ - |
+ python3 arrow/dev/tasks/crossbow.py \
+ --queue-path . \
+ --queue-remote {{ queue_remote_url }} \
+ upload-artifacts \
+ --pattern "arrow/python/repaired_wheels/*.whl" \
+ --sha {{ task.branch }} \
+ --tag {{ task.tag }}
+
+ {% if True or arrow.branch == 'master' %}
Review comment:
TODO: We should not forget to remove `True or` before we merge this.
##########
File path: dev/tasks/python-wheels/github.linux.amd64.yml
##########
@@ -95,7 +87,13 @@ jobs:
env:
CROSSBOW_GITHUB_TOKEN: {{ '${{ secrets.CROSSBOW_GITHUB_TOKEN }}' }}
- {% if arrow.branch == 'master' %}
+ {% if True or arrow.branch == 'master' %}
Review comment:
TODO: We should not forget to remove `True or` before we merge this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-797515456
@github-actions crossbow submit wheel-manylinux2014-*-arm64
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eladroz edited a comment on pull request #9285: ARROW-10349: [Python] Build and publish aarch64 wheels [WIP]
Posted by GitBox <gi...@apache.org>.
eladroz edited a comment on pull request #9285:
URL: https://github.com/apache/arrow/pull/9285#issuecomment-766086589
@xhochy @kszucs hi, I've published my experience with building on AWS arm64 VM's [here](https://lists.apache.org/thread.html/r274a6f9a3d0133fe3b247ea9bb1da6c6f24fc1511508d1b36ead0ae1%40%3Cuser.arrow.apache.org%3E), but once I've tried to added ARROW_S3 component I got the same issue as above
However, compiling the AWS SDK from source rather than through vcpkg does seem to work after some fighting with it. Is that an option? if so, I'll document the steps.
I've proceeding with buillding manually, to see what the next issues are:
* Turned `-DARROW_GANDIVA=OFF` (some issue with llvm for aarch64)
* Needed to copy OpenSSL (`libcrypto.a`) that was fetched by vcpkg to my `/usr/local/lib64`, instead of the one built when compiling the AWS SDK locally (which seems to have an issue with arm-related symbol `OPENSSL_armcap_P`)
* Did not attempt to turn on CUDA yet
Following the above -> Arrow C++ libs were built. Tests passed except for:
```
The following tests FAILED:
41 - arrow-s3fs-test (Failed)
43 - arrow-flight-test (Failed)
50 - arrow-python-test (Failed)
```
(didn't go into that yet)
Moving on to PyArrow, built with all components except `PYARROW_WITH_GANDIVA` and `_CUDA`:
* Building with `PYARROW_WITH_TENSORFLOW` - I got [this existing issue](https://github.com/apache/arrow/issues/7065) when running tests
* Turning off TF component, build passed.
* Tests passed, some are skipped:
`3329 passed, 351 skipped, 17 xfailed, 49 warnings in 53.40s`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org