You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/17 18:42:41 UTC
[GitHub] [arrow] lidavidm opened a new pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
lidavidm opened a new pull request #10348:
URL: https://github.com/apache/arrow/pull/10348
This significantly speeds up Python tests.
I opted to put it in Python instead of in the shell script so that interactive local development also benefits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10348:
URL: https://github.com/apache/arrow/pull/10348#issuecomment-842532124
Before:
```
================================================== slowest 10 durations ===================================================
159.74s call pyarrow/tests/test_fs.py::test_s3_proxy_options
15.99s call pyarrow/tests/test_dataset.py::test_open_dataset_from_s3_with_filesystem_uri
15.22s call pyarrow/tests/test_fs.py::test_s3_real_aws_region_selection
3.19s call pyarrow/tests/test_fs.py::test_s3_real_aws
3.12s call pyarrow/tests/test_dataset.py::test_write_dataset_s3
2.03s setup pyarrow/tests/test_fs.py::test_open_input_stream[S3FileSystem-gzip-256-compress]
2.02s call pyarrow/tests/test_fs.py::test_filesystem_is_functional_after_pickling[S3FileSystem]
2.02s setup pyarrow/tests/test_dataset.py::test_open_dataset_from_uri_s3
2.01s setup pyarrow/tests/test_dataset.py::test_open_dataset_from_uri_s3_fsspec
2.01s setup pyarrow/tests/test_dataset.py::test_write_dataset_s3
```
After:
```
================================================== slowest 10 durations ===================================================
0.69s call pyarrow/tests/test_fs.py::test_s3_real_aws_region_selection
0.63s call pyarrow/tests/test_fs.py::test_s3_options
0.31s call pyarrow/tests/parquet/test_dataset.py::test_read_multiple_files[True]
0.27s setup pyarrow/tests/test_dataset.py::test_make_fragment
0.25s call pyarrow/tests/test_dataset.py::test_open_dataset_from_uri_s3_fsspec
0.22s setup pyarrow/tests/test_dataset.py::test_open_dataset_from_uri_s3
0.21s call pyarrow/tests/parquet/test_dataset.py::test_read_partitioned_directory_s3fs[True]
0.19s call pyarrow/tests/test_dataset.py::test_union_dataset_from_other_datasets
0.18s call pyarrow/tests/parquet/test_dataset.py::test_read_partitioned_directory_s3fs[False]
0.16s call pyarrow/tests/parquet/test_dataset.py::test_read_multiple_files[False]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10348:
URL: https://github.com/apache/arrow/pull/10348#issuecomment-843116589
Merging this; looks like Homebrew has having issues causing the MacOS build to time out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm closed pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
lidavidm closed pull request #10348:
URL: https://github.com/apache/arrow/pull/10348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
lidavidm commented on pull request #10348:
URL: https://github.com/apache/arrow/pull/10348#issuecomment-842743117
Ah, I ran into this earlier somewhere else and Neal suggested it back then. Plus you can see this same env var scattered throughout the Arrow repository as people have run into it again and again in different contexts, so really someone just needed to throw it into Python.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10348:
URL: https://github.com/apache/arrow/pull/10348#issuecomment-842537237
https://issues.apache.org/jira/browse/ARROW-12810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #10348: ARROW-12810: [Python] Stop AWS SDK from looking for metadata service
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #10348:
URL: https://github.com/apache/arrow/pull/10348#issuecomment-842921493
Nice, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org