You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/08/18 22:42:24 UTC
[GitHub] [arrow] corleyma opened a new pull request #7997: ARROW-9266: [Python/Packaging] enable C++ S3FS in macOS wheels
corleyma opened a new pull request #7997:
URL: https://github.com/apache/arrow/pull/7997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-680090405
If you're going to try to get the bundled `build_awssdk` macro to work, a couple of notes:
1. The features enabled in the bundled ep (https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2635) don't match what is actually required (https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L2681-L2685) so you'll need to fix that
2. If it doesn't build, perhaps look to https://github.com/TileDB-Inc/TileDB/blob/dev/cmake/Modules/FindAWSSDK_EP.cmake for inspiration since they manage to build it
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679470592
@github-actions crossbow submit wheel-osx-*
Oh, sorry. I didn't know that we have the https://github.com/apache/arrow/blob/master/dev/archery/archery/bot.py#L180-L183 check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-680040081
We experimented a bit with conda-press a while back but it yielded poor results for us (wheels that were _much_ larger than our current ones). I expect we are going to be fighting to keep our wheels at an acceptable size for the foreseeable future (at some point we need to try to break up the Python project into multiple interdependent wheels to enable modular installations)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679370468
FWIW the latest Travis failures do look like https://github.com/aws/aws-sdk-cpp/issues/1309. I don't think Homebrew is a viable source for aws-sdk-cpp for us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-680040870
@github-actions crossbow submit wheel-osx-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679350519
I don't remember the specifics, but IIRC the AWS SDK build procedure always takes up the system version of OpenSSL, while in many cases (especially packaging-related) we want to use a dedicated OpenSSL version (such as Conda-provided version when building a conda package, etc.).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679333928
@pitrou It seems like it might be worth digging into how to fix/enable builds of AWS SDK in the `ThirdpartyToolchain.cmake`. Looks like you disabled it [here](https://github.com/corleyma/arrow/commit/7ec173172953f6ab27f7bad495bd49bff451d03c#diff-2420b0c5b6bdad921f1d538f92d64b59R2388).
Is there any additional context you could provide re: the libcrypto linking problem so I can try to dig into this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679418775
>So any C++ dependency that's expressed naturally in conda (as a dependency on just another package) has to become statically bundled in a Python wheel.
Bundling those dependencies is exactly what I was hoping conda would do when building a wheel. I guess that's probably being a bit optimistic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7997: ARROW-9266: [Python/Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-675789132
@github-actions crossbow submit wheel-osx-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679409935
So I maintain that building AWS SDK from source during the wheel build process is probably the most reasonable way forward here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7997: ARROW-9266: [Python/Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#discussion_r472571654
##########
File path: dev/tasks/python-wheels/travis.osx.yml
##########
@@ -21,6 +21,7 @@ language: generic
addons:
homebrew:
packages:
+ - aws-sdk-cpp
Review comment:
Could you keep this list in alphabetical order?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-703560812
We'll complete this by #8315.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679435086
@kou Ah, interesting, good to know! I could not seem to get `@github-actions crossbow submit -g conda` to work as expected in a different PR, but perhaps it's just the `-g` option that isn't supported?
@pitrou It looks like there is something called [conda-press](https://github.com/conda-incubator/conda-press) that may be closer to what I was imagining, but it doesn't seem very mature yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679431464
Revision: 0b3365dbee909715c7fa7dbdaeebac50357f72d2
Submitted crossbow builds: [ursa-labs/crossbow @ actions-497](https://github.com/ursa-labs/crossbow/branches/all?query=actions-497)
|Task|Status|
|----|------|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou closed pull request #7997:
URL: https://github.com/apache/arrow/pull/7997
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679406134
@kou I pushed some changes which optimistically re-enable building of AWS C++ sdk from source.
@pitrou As said before, I'm not super familiar with conda but a cursory glance at the docs implies it [might be possible to build wheels from conda](https://docs.conda.io/projects/conda-build/en/latest/resources/define-metadata.html?highlight=wheel#output-type)? I wonder if that might be a good approach for the macOS wheels.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679357816
Revision: e2359f53f4f053ea9181963d3859d7d43c5dee04
Submitted crossbow builds: [ursa-labs/crossbow @ actions-496](https://github.com/ursa-labs/crossbow/branches/all?query=actions-496)
|Task|Status|
|----|------|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-680053835
Revision: e7948bed71cc81ee2f25f8a01ebf3c383e1fac9c
Submitted crossbow builds: [ursa-labs/crossbow @ actions-498](https://github.com/ursa-labs/crossbow/branches/all?query=actions-498)
|Task|Status|
|----|------|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679430872
@github-actions crossbow submit wheel-osx-*
FYI: Anybody can run jobs to build wheels for macOS by writing the above comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] nealrichardson commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
nealrichardson commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-676607641
So this is failing because it's trying to build the bundled aws-sdk-cpp external project, but that's not implemented: https://travis-ci.org/github/ursa-labs/crossbow/builds/719137336#L1231-L1232
You could try adding `-DAWSSDK_SOURCE=SYSTEM` to try to pick up the Homebrew package. I suspect that will also fail due to https://github.com/aws/aws-sdk-cpp/issues/1309 but we'll see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-675906606
@nealrichardson
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679458248
@github-actions crossbow submit wheel-osx-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7997: ARROW-9266: [Python/Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-675756491
https://issues.apache.org/jira/browse/ARROW-9266
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679350747
That was with an older AWS SDK version some time ago, though, so you could try to enable it and see what happens now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-675789705
Revision: ad77b8fdacb942cee7556e20b09fc8dcf82c5e1a
Submitted crossbow builds: [ursa-labs/crossbow @ actions-493](https://github.com/ursa-labs/crossbow/branches/all?query=actions-493)
|Task|Status|
|----|------|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679461439
@kou seems crossbow builds won't run for me, alas. 😿
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679409666
I can't answer definitely, but I'm extremely skeptical that conda-build will be able to proper usable Python wheels in anything other than the most simple cases, and Arrow is really at the other end of the complexity spectrum when it comes to dependencies and packaging.
To understand why, you have to understand that Python wheels are not able to express non-Python dependencies, while conda packages are. So any C++ dependency that's expressed naturally in conda (as a dependency on just another package) has to become statically bundled in a Python wheel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
kou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679356761
@github-actions crossbow submit wheel-osx-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] corleyma commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
corleyma commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679408850
> The approach we use for manylinux wheels is to build the AWS SDK from source separately, and it works. Why would it be different on macOS?
I'm not sure if it would be possible to use conda to build manylinux-compatible wheels, but in general it seems that the conda builds of pyarrow are more supported than the wheels. I wonder if using conda-build to generate wheels would reduce the lag time for new functionality to make it into the wheels and reduce overhead for maintainers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7997: ARROW-9266: [Python][Packaging] enable C++ S3FS in macOS wheels
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7997:
URL: https://github.com/apache/arrow/pull/7997#issuecomment-679407504
The approach we use for manylinux wheels is to build the AWS SDK from source separately, and it works. Why would it be different on macOS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org