You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/10/19 18:36:03 UTC
[GitHub] [arrow] AWSjswinney opened a new pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
AWSjswinney opened a new pull request #8491:
URL: https://github.com/apache/arrow/pull/8491
add support for building python aarch64 wheels for linux
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] RafayAK commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
RafayAK commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-753345204
So, PyArrow takes a colossal amount of time when building for `aarch64`, probably because it building from sources each time, and running the build in a CI with emulation of ARM takes even longer. Can I help in any way to fast-track this issue, since it seems like it's been dormant for a while?
for reference when building an Airflow container for `aarch64` (5 hrs in an still at it):
```
Collecting pyarrow<3.0dev,>=1.0.0
Downloading pyarrow-2.0.0.tar.gz (58.9 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] RafayAK edited a comment on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
RafayAK edited a comment on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-753345204
So, PyArrow takes a colossal amount of time when building for `aarch64`, probably because it building from sources each time, and running the build in a CI with emulation of ARM takes even longer. Can I help in any way to fast-track this issue, since it seems like it's been dormant for a while?
for reference when building an Airflow container for `aarch64` (5 hrs in and still at it):
```
Collecting pyarrow<3.0dev,>=1.0.0
Downloading pyarrow-2.0.0.tar.gz (58.9 MB)
Installing build dependencies: started
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
Installing build dependencies: still running...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-717796605
Yes, going to take a look today.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712367868
You'll need to create a Travis-CI job that runs this docker-compose entry.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AWSjswinney commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
AWSjswinney commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712367815
This change makes it possible to build aarch64 python wheels, but I haven't figured out how to modify the CI code to use Travis CI the do Arm builds. Travis-CI.com is the only CI currently in use by Apache Arrow that supports Arm64 builds on linux. Guidance on the necessary changes to CI/Travis code would be appreciated.
I assume that something [here](https://github.com/apache/arrow/tree/master/dev/tasks/python-wheels) needs to be modified, but it's not clear to me how Travis-CI is invoked with this yaml file or where I would need to add code to add an additional yaml file, `travis.linux.aarc64.yml`, for example. Guidance or recommendations are appreciated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712373502
@xhochy @kszucs Can you help with this? Our conventions are getting in the way here (e.g. `ARCH` is `amd64` rather than `x86_64`, we use a specific hash for the upstream manylinux base image...).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] yoonsikp commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
yoonsikp commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-790429618
Would truly appreciate having aarch64 wheel support for apache arrow!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-717856257
I'm currently pruning old crossbow branches and tags since drone doesn't pick up the aarch64 builds and the github settings page can't be loaded to investigate the issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-742493941
A possible way to unblock this would also be just running the `aarch64` container with emulation on Github Actions or Azure Pipelines. This will take 3-5x the time but it will at least run!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-792364067
We should favor working on https://github.com/apache/arrow/pull/9285 using vcpkg.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712563608
We have some self-hosted aarch64 machines that we could use to run these, too, at some point
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] RafayAK commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
RafayAK commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-753346763
Oh, just noticed @AWSjswinney has a pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-717856257
I'm currently pruning old crossbow branches and tags since drone doesn't pick up the aarch64 builds and the github settings page can't be loaded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] aitor94 commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
aitor94 commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-778135205
I'm trying to install pyarrow in a m6g instance on AWS and I can't. This PR may help to solve my issue. [https://stackoverflow.com/questions/64928357/how-to-install-pyarrow-in-amazon-linux-2-m6g-instance?noredirect=1#comment116973612_64928357](stack_overflow)
Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712711653
Crossbow also support using drone.io for running `aarch64` builds. That could be an alternative to Travis.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AWSjswinney commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
AWSjswinney commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-752707138
I believe this PR now has everything needed to build and publish aarch64 wheels for manylinux2014. Thanks for reviewing it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-717672056
@AWSjswinney it looks like this needs a rebase. @kszucs do you have time to review, when that is done?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs closed pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs closed pull request #8491:
URL: https://github.com/apache/arrow/pull/8491
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] AWSjswinney commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
AWSjswinney commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-754123351
@RafayAK It isn't exactly quick with this change but this test build I ran completed in about 35 minutes: https://travis-ci.com/github/AWSjswinney/crossbow/builds/211319456
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8491: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-799292575
Closing in favor of #9285
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #8491: WIP: ARROW-10349: [Python] linux aarch64 wheels
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8491:
URL: https://github.com/apache/arrow/pull/8491#issuecomment-712392450
I'd suggest to expand the docker-compose configuration on short term until we can figure out a nicer way to map different arch naming conventions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org