You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/06/02 22:34:38 UTC
[GitHub] [arrow] wesm opened a new pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
wesm opened a new pull request #7334:
URL: https://github.com/apache/arrow/pull/7334
Current manylinux wheel packages on master:
* .whl manylinux1 package is **61 MB**
* Installed size is **223 MB**
This patch
* .whl package is **15 MB**
* Installed size is **57 MB**
That's more than a 4x size reduction. There's several things in this patch:
* We no longer ship 2 copies of shared libraries in the wheels. We ship just the SO-versioned shared libraries now. Because this creates problems for linkers (`-larrow -lparquet` etc won't work as is), I added function that tries to create the necessary symlinks when you call `pyarrow.get_library_dirs()`. If pyarrow is installed somewhere where you can't create symlinks and the symlinks don't exist, it will print a message instructing you to run the symlinking function as root. This was the simplest strategy I could think to get out of this mess.
* Gandiva is disabled. If we're going to ship Gandiva as a wheel, I think we should do it as an add-on `pyarrow_gandiva` package per ARROW-8518.
* Environment variable PYARROW_INSTALL_TESTS added to not install `pyarrow.tests`, which is about 2.3MB uncompressed. I don't think we need to ship the tests in the wheels.
* Compiled Cython sources are no longer shipped.
I'll need some help kicking the tires on macOS and Windows and to make sure the Crossbow builds are all passing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638427234
I made the `pa.create_library_symlinks()` an explicit opt-in. I'm wondering if we should add a test to make sure that building a C extension against the installed wheel works so this doesn't get broken in the future
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638428377
Sounds fine to me, we can open a JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-639547670
@xhochy could you please give it a review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638160267
> I'd like to keep them. It's helpful to test if an installation works correctly.
Couldn't we solve this problem another way? If a user wants to run the whole test suite locally they need more than just the pyarrow/tests directory. I don't think it's worth bloating the installs for an infrequent use case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-637846927
https://issues.apache.org/jira/browse/ARROW-5082
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638168355
> > I added function that tries to create the necessary symlinks when you call pyarrow.get_library_dirs()
>
> It's a bad idea to add side effects to a simple inquiry function. This should IMHO be in a separate function (e.g. `create_library_symlinks`).
Agree with Antoine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm closed pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm closed pull request #7334:
URL: https://github.com/apache/arrow/pull/7334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638709929
@github-actions crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638169226
> @kszucs can you help me get this across the finish line?
Yes, I'll ensure that the wheel packaging builds work properly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-639019570
@github-actions crossbow submit wheel-manylinux*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm edited a comment on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638414377
> Well, I won't argue too much about it. But at some point we had decided that wheels were too much of a burden for us, and now it seems we're going out of our way to please people. I'm not sure I understand the strategy.
Well, the wheels are being installed [at least 6.5 million times per month](https://pypistats.org/packages/pyarrow) (for point of reference, for pandas it's 22.7M) and so wheel use has an impact on the health and success of the open source project. My attitude is that we shouldn't feel too bad about "taking things away" from the wheels absent more enthusiastic maintainers. With a half day's labor I was able to shrink the wheels by 4x -- the tests / no tests thing wasn't the most significant change but I definitely don't want to be going out of our way to put things in the wheels or maintain special code to cater to wheel users under the present circumstances. I think at least this will stymie some of the pain for some period of time until perhaps more maintainers come out of the woodwork (or I can afford to recruit and hire them).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-657704528
@francisco-hoo For now this is only available in nightly builds:
https://arrow.apache.org/docs/python/install.html#installing-nightly-packages
Soon we will release a 1.0.0 with those improvements included, though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638385825
@github-actions crossbow submit wheel-osx-*
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638315649
> > from the packages (I'd rather keep them though) please defer it to another pull request because we need to update more CI scripts.
>
> I'm fine with debating whether to ship the tests separately.
>
> The people who benefit from being able to do `pytest --pyargs pyarrow` are primarily the developers (i.e. us), not the users. If we want to enable users to run the test suite locally on their machine, we should IMHO instead implement a function that downloads the tests along with the test dependencies (e.g. the testing data repos) and then executes them. If say < 0.1% of users ever need to do this, why should > 99.9% of users bear the burden?
Hard to disagree with that argument. Either way I'm deferring it to a follow-up because it'll involve quite some CI and packaging updates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm edited a comment on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638324065
> It's between the burden for the user of two additional megabytes installed, vs. the burden for us of
Well, the point is that we don't _have_ to do it at all. At no point in the 52 months since Apache Arrow started do I recall a user running the test suite out of a wheel or asking about doing so. If this is truly something that people need to be able to do, maybe an interested party can contribute it to the project? We've already expressed that we are going to limit our investment of time in maintaining wheels, and, from what I can tell, smaller wheels -> fewer complaints (here a 3% savings isn't that compelling but given that people are trying to squeeze pyarrow into deployments in AWS lambda that have to be < 250 MB including all dependencies, every megabyte does indeed count).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-639898609
Merging. Thanks all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on a change in pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#discussion_r434773060
##########
File path: cpp/cmake_modules/ThirdpartyToolchain.cmake
##########
@@ -707,15 +707,14 @@ endif()
# - Gandiva has a compile-time (header-only) dependency on Boost, not runtime.
# - Tests need Boost at runtime.
# - S3FS and Flight benchmarks need Boost at runtime.
+
if(ARROW_BUILD_INTEGRATION
OR ARROW_BUILD_TESTS
OR (ARROW_FLIGHT AND ARROW_BUILD_BENCHMARKS)
OR (ARROW_S3 AND ARROW_BUILD_BENCHMARKS)
OR ARROW_GANDIVA
OR (ARROW_WITH_THRIFT AND Thrift_SOURCE STREQUAL "BUNDLED")
- OR (ARROW_PARQUET
- AND CMAKE_CXX_COMPILER_ID STREQUAL "GNU"
- AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS "4.9"))
+ OR ARROW_PARQUET)
Review comment:
```
In file included from /usr/local/include/thrift/TApplicationException.h:23,
from /arrow/cpp/src/parquet/thrift_internal.h:36,
from /arrow/cpp/src/parquet/column_reader.cc:47:
/usr/local/include/thrift/Thrift.h:45:10: fatal error: boost/utility/enable_if.hpp: No such file or directory
#include <boost/utility/enable_if.hpp>
```
Seems like the parquet headers require boost headers as a transitive dependency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638318290
> If say < 0.1% of users ever need to do this, why should > 99.9% of users bear the burden?
It's between the burden for the user of two additional megabytes installed, vs. the burden for us of "implement [and maintain] a function that downloads the tests along with the test dependencies (e.g. the testing data repos) and then executes them".
For me it's a no-brainer to ship the tests with the wheels, but YMMV.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638411864
Revision: 271b25b2ff83000e89a4fdd24d7b6594383ba333
Submitted crossbow builds: [ursa-labs/crossbow @ actions-287](https://github.com/ursa-labs/crossbow/branches/all?query=actions-287)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-287-azure-wheel-manylinux2014-cp38)|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp36m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp37m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp38|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638258958
Revision: 2d89b44eb17701171618c87ece2ad8ac1e8bbe42
Submitted crossbow builds: [ursa-labs/crossbow @ actions-284](https://github.com/ursa-labs/crossbow/branches/all?query=actions-284)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-284-azure-wheel-manylinux2014-cp38)|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp36m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp37m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp38|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
xhochy commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638688588
> I would say that whoever depends on that may want to add the required unit test.
FYI @fjetter
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638428131
I would say that whoever depends on that may want to add the required unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638059015
> I added function that tries to create the necessary symlinks when you call pyarrow.get_library_dirs()
It's a bad idea to add side effects to a simple inquiry function. This should IMHO be in a separate function (e.g. `create_library_symlinks`).
> I don't think we need to ship the tests in the wheels.
I'd like to keep them. It's helpful to test if an installation works correctly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs edited a comment on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638315649
> I'm fine with debating whether to ship the tests separately.
>
> The people who benefit from being able to do `pytest --pyargs pyarrow` are primarily the developers (i.e. us), not the users. If we want to enable users to run the test suite locally on their machine, we should IMHO instead implement a function that downloads the tests along with the test dependencies (e.g. the testing data repos) and then executes them. If say < 0.1% of users ever need to do this, why should > 99.9% of users bear the burden?
Hard to disagree with that argument :)
Either way I'm deferring it to a follow-up because it'll involve quite some CI and packaging updates.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638414377
> Well, I won't argue too much about it. But at some point we had decided that wheels were too much of a burden for us, and now it seems we're going out of our way to please people. I'm not sure I understand the strategy.
Well, the wheels are being installed [at least 6.5 million times per month](https://pypistats.org/packages/pyarrow) and so wheel use has an impact on the health and success of the open source project. My attitude is that we shouldn't feel too bad about "taking things away" from the wheels absent more enthusiastic maintainers. With a half day's labor I was able to shrink the wheels by 4x -- the tests / no tests thing wasn't the most significant change but I definitely don't want to be going out of our way to put things in the wheels or maintain special code to cater to wheel users under the present circumstances. I think at least this will stymie some of the pain for some period of time until perhaps more maintainers come out of the woodwork (or I can afford to recruit and hire them).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638710771
Revision: 599660ef7deb5bc5c2aab831c90794e580ce8f3f
Submitted crossbow builds: [ursa-labs/crossbow @ actions-288](https://github.com/ursa-labs/crossbow/branches/all?query=actions-288)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-288-azure-wheel-manylinux2014-cp38)|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp36m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp37m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp38|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-637844597
@kszucs can you help me get this across the finish line?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638242809
Revision: 208bd8cb420ff53db554a136bb11471b2a00cfec
Submitted crossbow builds: [ursa-labs/crossbow @ actions-283](https://github.com/ursa-labs/crossbow/branches/all?query=actions-283)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-283-azure-wheel-manylinux2014-cp38)|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp36m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp37m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp38|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638384395
Revision: 00d9c864f94679bd2850a558019ebf645997e593
Submitted crossbow builds: [ursa-labs/crossbow @ actions-285](https://github.com/ursa-labs/crossbow/branches/all?query=actions-285)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-285-azure-wheel-manylinux2014-cp38)|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp36m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp37m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
|wheel-win-cp38|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638386729
Revision: 8dacdc99b642de75f5f14be6556a1307c783bfc4
Submitted crossbow builds: [ursa-labs/crossbow @ actions-286](https://github.com/ursa-labs/crossbow/branches/all?query=actions-286)
|Task|Status|
|----|------|
|wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
|wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638288928
> from the packages (I'd rather keep them though) please defer it to another pull request because we need to update more CI scripts.
I'm fine with debating whether to ship the tests separately.
The people who benefit from being able to do `pytest --pyargs pyarrow` are primarily the developers (i.e. us), not the users. If we want to enable users to run the test suite locally on their machine, we should IMHO instead implement a function that downloads the tests along with the test dependencies (e.g. the testing data repos) and then executes them. If say < 0.1% of users ever need to do this, why should > 99.9% of users bear the burden?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] xhochy commented on a change in pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
xhochy commented on a change in pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#discussion_r436015255
##########
File path: dev/tasks/python-wheels/win-build.bat
##########
@@ -96,9 +101,10 @@ set ARROW_TEST_DATA=%ARROW_SRC%\testing\data
%PYTHON_INTERPRETER% -c "import pyarrow" || exit /B
%PYTHON_INTERPRETER% -c "import pyarrow.parquet" || exit /B
%PYTHON_INTERPRETER% -c "import pyarrow.flight" || exit /B
-%PYTHON_INTERPRETER% -c "import pyarrow.gandiva" || exit /B
%PYTHON_INTERPRETER% -c "import pyarrow.dataset" || exit /B
+@rem %PYTHON_INTERPRETER% -c "import pyarrow.gandiva" || exit /B
Review comment:
Just delete this line
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638383478
@github-actions crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638410784
@github-actions crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638404959
Well, I won't argue too much about it. But at some point we had decided that wheels were too much of a burden for us, and now it seems we're going out of our way to please people. I'm not sure I understand the strategy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-639020290
Revision: 83b0c5f977d8a8d956ae1b0c3d1298ae29c6b5ba
Submitted crossbow builds: [ursa-labs/crossbow @ actions-289](https://github.com/ursa-labs/crossbow/branches/all?query=actions-289)
|Task|Status|
|----|------|
|wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux1-cp35m)|
|wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux1-cp36m)|
|wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux1-cp37m)|
|wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux1-cp38)|
|wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2010-cp35m)|
|wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2010-cp36m)|
|wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2010-cp37m)|
|wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2010-cp38)|
|wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2014-cp35m)|
|wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2014-cp36m)|
|wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2014-cp37m)|
|wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-289-azure-wheel-manylinux2014-cp38)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] francisco-hoo commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
francisco-hoo commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-657701553
How can I download this ~15MB .whl version of PyArrow?
Just find the `pyarrow-0.17.1-cp38-cp38-manylinux2014_x86_64.whl (63.8 MB)` version.
Due to AWS Lambda-Layers limitations, this pull is essential... Thanks all!!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638241816
@github-actions crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638167989
At first I didn't like that the tests are shipped with the packages, but later on I found it useful. It also worth mentioning that many of our packaging builds and CI tests run the pyarrow unittests using `pytest --pyargs pyarrow` after installation.
If we decide to remove the tests from the packages (I'd rather keep them though) please defer it to another pull request because we need to update more CI scripts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638257730
@github-actions crossbow submit -g wheel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm edited a comment on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm edited a comment on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638428377
Sounds fine to me, we can open a JIRA.
EDIT: https://issues.apache.org/jira/browse/ARROW-9033
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7334: ARROW-5082: [Python] Substantially reduce Python wheel package and install size
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7334:
URL: https://github.com/apache/arrow/pull/7334#issuecomment-638324065
> It's between the burden for the user of two additional megabytes installed, vs. the burden for us of
Well, the point is that we don't _have_ to do it at all. At no point in the 52 months since Apache Arrow started do I recall a user running the test suite out of a wheel or asking about doing so. If this is truly something that people need to be able to do, maybe an interested party can contribute it to the project? We've already expressed that we are going to limit our investment of time in maintaining wheels, and, from what I can tell, smaller wheels -> fewer complaints (here a 3% savings isn't that compelling but given that people are trying to squeeze pyarrow into deployments in AWS lambda that have to be < 250 MB, every megabyte does indeed count).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org