You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/11/23 11:45:52 UTC
[GitHub] [arrow] milesgranger opened a new pull request, #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
milesgranger opened a new pull request, #14716:
URL: https://github.com/apache/arrow/pull/14716
Will fix [ARROW-18392](https://issues.apache.org/jira/browse/ARROW-18392)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1325117757
The pandas-0.24 job failure is expected to fail and should be fixed on: https://github.com/apache/arrow/pull/14714
The pandas nightly and upstream_devel failures are unrelated and I think should be fixed on its own ticket. I have opened this ticket to track them: https://issues.apache.org/jira/browse/ARROW-18394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1325082099
@github-actions crossbow submit test-conda-python-*-pandas-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; set anonymous=True
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1327580103
Benchmark runs are scheduled for baseline = 7a47e8dcedd2f3198df90d68650a5e9b5597bb58 and contender = ade42669e44918e89b862610e79ac36c0cca590d. ade42669e44918e89b862610e79ac36c0cca590d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/c11b6eb36da749589f05f17d1ee169ee...ac4e3a3d1fc84d6cb08a00ffc6051283/)
[Finished :arrow_down:0.13% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/ef66cc28f13142a2a7609895d3955c3b...8cce597ca5fc448eb42ef998fdfd3692/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/3f72ecd4b6d14a2ebd25894831d5a624...3505ac25717f4058960cf812c853ae91/)
[Finished :arrow_down:0.1% :arrow_up:0.07%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/2bdc259e359e47cfb8a2674b51ca4a5c...0e010b2efe4b4ad0806b0dcbffb9ca1d/)
Buildkite builds:
[Finished] [`ade42669` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1921)
[Finished] [`ade42669` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1943)
[Finished] [`ade42669` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1913)
[Finished] [`ade42669` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1935)
[Finished] [`7a47e8dc` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1920)
[Finished] [`7a47e8dc` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1942)
[Finished] [`7a47e8dc` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1912)
[Finished] [`7a47e8dc` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1934)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1324933929
@github-actions crossbow submit test-conda-python-*-pandas-*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1324932357
https://issues.apache.org/jira/browse/ARROW-18392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] assignUser commented on a diff in pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
assignUser commented on code in PR #14716:
URL: https://github.com/apache/arrow/pull/14716#discussion_r1030444406
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -1321,22 +1321,23 @@ def test_s3fs_wrong_region():
from pyarrow.fs import S3FileSystem
# wrong region for bucket
- fs = S3FileSystem(region='eu-north-1')
+ fs = S3FileSystem(region='eu-north-1', anonymous=True)
Review Comment:
I'd add a comment explaining why this is important. Do we have other tests s3 tests that could also need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] milesgranger commented on a diff in pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
milesgranger commented on code in PR #14716:
URL: https://github.com/apache/arrow/pull/14716#discussion_r1030526675
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -1321,22 +1321,24 @@ def test_s3fs_wrong_region():
from pyarrow.fs import S3FileSystem
# wrong region for bucket
- fs = S3FileSystem(region='eu-north-1')
+ # anonymous=True incase CI/etc has invalid credentials
+ fs = S3FileSystem(region='eu-north-1', anonymous=True)
msg = ("When getting information for bucket 'voltrondata-labs-datasets': "
r"AWS Error UNKNOWN \(HTTP status 301\) during HeadBucket "
"operation: No response body. Looks like the configured region is "
- "'eu-north-1' while the bucket is located in 'us-east-2'."
- "|NETWORK_CONNECTION")
- with pytest.raises(OSError, match=msg) as exc:
+ "'eu-north-1' while the bucket is located in 'us-east-2'.")
+ with pytest.raises(OSError) as exc:
fs.get_file_info("voltrondata-labs-datasets")
- # Sometimes fails on unrelated network error, so next call would also fail.
- if 'NETWORK_CONNECTION' in str(exc.value):
+ # Sometimes fails on unrelated network or permission error, so next
+ # call would also fail.
+ if 'UNKNOWN' not in str(exc.value):
Review Comment:
Ya, good point. Since the main issue was `anonymous=True` needing to be set, I'll reset everything else.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou merged pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; set anonymous=True
Posted by GitBox <gi...@apache.org>.
pitrou merged PR #14716:
URL: https://github.com/apache/arrow/pull/14716
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] milesgranger commented on a diff in pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
milesgranger commented on code in PR #14716:
URL: https://github.com/apache/arrow/pull/14716#discussion_r1030463465
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -1321,22 +1321,23 @@ def test_s3fs_wrong_region():
from pyarrow.fs import S3FileSystem
# wrong region for bucket
- fs = S3FileSystem(region='eu-north-1')
+ fs = S3FileSystem(region='eu-north-1', anonymous=True)
Review Comment:
Done (https://github.com/apache/arrow/pull/14716/commits/2342d3068d23754926fa75957a5e6324a7884836), and I found only one other test that uses a real AWS call, aptly named `test_s3_real_aws` but that already sets `anonymous=True`. All others appear to use the endpoint override / minio.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14716:
URL: https://github.com/apache/arrow/pull/14716#issuecomment-1325084854
Revision: cadef87b204f37ef15838dc6d7092b3b7d446657
Submitted crossbow builds: [ursacomputing/crossbow @ actions-2fbd3611ff](https://github.com/ursacomputing/crossbow/branches/all?query=actions-2fbd3611ff)
|Task|Status|
|----|------|
|test-conda-python-3.7-pandas-0.24|[](https://github.com/ursacomputing/crossbow/actions/runs/3532564014/jobs/5927070171)|
|test-conda-python-3.7-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/3532563248/jobs/5927068211)|
|test-conda-python-3.8-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/3532562802/jobs/5927067114)|
|test-conda-python-3.8-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/3532562061/jobs/5927065343)|
|test-conda-python-3.9-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/3532562477/jobs/5927066168)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] milesgranger commented on a diff in pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
milesgranger commented on code in PR #14716:
URL: https://github.com/apache/arrow/pull/14716#discussion_r1030463465
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -1321,22 +1321,23 @@ def test_s3fs_wrong_region():
from pyarrow.fs import S3FileSystem
# wrong region for bucket
- fs = S3FileSystem(region='eu-north-1')
+ fs = S3FileSystem(region='eu-north-1', anonymous=True)
Review Comment:
Done, and I found only one other test that uses a real AWS call, aptly named `test_s3_real_aws` but that already sets `anonymous=True`. All others appear to use the endpoint override / minio.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #14716: ARROW-18392: [Python] Fix test_s3fs_wrong_region; match when UNKNOWN
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #14716:
URL: https://github.com/apache/arrow/pull/14716#discussion_r1030523191
##########
python/pyarrow/tests/test_fs.py:
##########
@@ -1321,22 +1321,24 @@ def test_s3fs_wrong_region():
from pyarrow.fs import S3FileSystem
# wrong region for bucket
- fs = S3FileSystem(region='eu-north-1')
+ # anonymous=True incase CI/etc has invalid credentials
+ fs = S3FileSystem(region='eu-north-1', anonymous=True)
msg = ("When getting information for bucket 'voltrondata-labs-datasets': "
r"AWS Error UNKNOWN \(HTTP status 301\) during HeadBucket "
"operation: No response body. Looks like the configured region is "
- "'eu-north-1' while the bucket is located in 'us-east-2'."
- "|NETWORK_CONNECTION")
- with pytest.raises(OSError, match=msg) as exc:
+ "'eu-north-1' while the bucket is located in 'us-east-2'.")
+ with pytest.raises(OSError) as exc:
fs.get_file_info("voltrondata-labs-datasets")
- # Sometimes fails on unrelated network error, so next call would also fail.
- if 'NETWORK_CONNECTION' in str(exc.value):
+ # Sometimes fails on unrelated network or permission error, so next
+ # call would also fail.
+ if 'UNKNOWN' not in str(exc.value):
Review Comment:
Why "not in"? Don't we want this condition to be a bit more specific?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org