You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2025/12/17 08:33:38 UTC
[I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
raulcd opened a new issue, #48565:
URL: https://github.com/apache/arrow/issues/48565
### Describe the bug, including details regarding any error messages, version, and platform.
It seems that the following bucket `s3://mf-nwp-models/README.txt` used on a couple of tests is not accessible anymore:
```
=================================== FAILURES ===================================
______________________ test_s3_real_aws_region_selection _______________________
@pytest.mark.s3
def test_s3_real_aws_region_selection():
# Taken from a registry of open S3-hosted datasets
# at https://github.com/awslabs/open-data-registry
fs, path = FileSystem.from_uri('s3://mf-nwp-models/README.txt')
assert fs.region == 'eu-west-1'
> with fs.open_input_stream(path) as f:
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_fs.py:1935:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyarrow/_fs.pyx:857: in pyarrow._fs.FileSystem.open_input_stream
???
pyarrow/error.pxi:155: in pyarrow.lib.pyarrow_internal_check_status
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: When reading information for key 'README.txt' in bucket 'mf-nwp-models': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
pyarrow/error.pxi:92: OSError
_______________________________ test_s3_finalize _______________________________
@pytest.mark.s3
def test_s3_finalize():
# Once finalize_s3() was called, most/all operations on S3 filesystems
# should raise.
code = """if 1:
import pytest
from pyarrow.fs import (FileSystem, S3FileSystem,
ensure_s3_initialized, finalize_s3)
fs, path = FileSystem.from_uri('s3://mf-nwp-models/README.txt')
assert fs.region == 'eu-west-1'
f = fs.open_input_stream(path)
f.read(50)
finalize_s3()
with pytest.raises(ValueError, match="S3 .* finalized"):
f.read(50)
with pytest.raises(ValueError, match="S3 .* finalized"):
fs.open_input_stream(path)
with pytest.raises(ValueError, match="S3 .* finalized"):
S3FileSystem(anonymous=True)
with pytest.raises(ValueError, match="S3 .* finalized"):
FileSystem.from_uri('s3://mf-nwp-models/README.txt')
"""
> subprocess.check_call([sys.executable, "-c", code])
opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_fs.py:2104:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
popenargs = (['/opt/conda/envs/arrow/bin/python3.10', '-c', 'if 1:\n import pytest\n from pyarrow.fs import (FileSys...ValueError, match="S3 .* finalized"):\n FileSystem.from_uri(\'s3://mf-nwp-models/README.txt\')\n '],)
kwargs = {}, retcode = 1
cmd = ['/opt/conda/envs/arrow/bin/python3.10', '-c', 'if 1:\n import pytest\n from pyarrow.fs import (FileSyst...s(ValueError, match="S3 .* finalized"):\n FileSystem.from_uri(\'s3://mf-nwp-models/README.txt\')\n ']
def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.
The arguments are the same as for the call function. Example:
check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E subprocess.CalledProcessError: Command '['/opt/conda/envs/arrow/bin/python3.10', '-c', 'if 1:\n import pytest\n from pyarrow.fs import (FileSystem, S3FileSystem,\n ensure_s3_initialized, finalize_s3)\n\n fs, path = FileSystem.from_uri(\'s3://mf-nwp-models/README.txt\')\n assert fs.region == \'eu-west-1\'\n f = fs.open_input_stream(path)\n f.read(50)\n\n finalize_s3()\n\n with pytest.raises(ValueError, match="S3 .* finalized"):\n f.read(50)\n with pytest.raises(ValueError, match="S3 .* finalized"):\n fs.open_input_stream(path)\n with pytest.raises(ValueError, match="S3 .* finalized"):\n S3FileSystem(anonymous=True)\n with pytest.raises(ValueError, match="S3 .* finalized"):\n FileSystem.from_uri(\'s3://mf-nwp-models/README.txt\')\n ']' returned non-zero exit status 1.
opt/conda/envs/arrow/lib/python3.10/subprocess.py:369: CalledProcessError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
File "<string>", line 8, in <module>
File "pyarrow/_fs.pyx", line 857, in pyarrow._fs.FileSystem.open_input_stream
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: When reading information for key 'README.txt' in bucket 'mf-nwp-models': AWS Error ACCESS_DENIED during HeadObject operation: No response body.
```
This has failed on several nightly jobs, I haven't double checked all of them but at least the first ones are due to the same issue:
- [test-conda-python-3.10](https://github.com/ursacomputing/crossbow/actions/runs/20287540566/job/58264656014)
- [test-conda-python-3.10-pandas-1.3.4-numpy-1.21.2](https://github.com/ursacomputing/crossbow/actions/runs/20287538590/job/58264649982)
- [test-conda-python-3.11](https://github.com/ursacomputing/crossbow/actions/runs/20287539559/job/58264652979)
- [test-conda-python-3.11-pandas-latest-numpy-latest](https://github.com/ursacomputing/crossbow/actions/runs/20287537486/job/58264646041)
- [test-conda-python-3.12](https://github.com/ursacomputing/crossbow/actions/runs/20287537454/job/58264646004)
- [test-conda-python-3.12-pandas-latest-numpy-1.26](https://github.com/ursacomputing/crossbow/actions/runs/20287538710/job/58264650418)
- [test-conda-python-3.12-pandas-latest-numpy-latest](https://github.com/ursacomputing/crossbow/actions/runs/20287540482/job/58264655747)
- [test-conda-python-3.13](https://github.com/ursacomputing/crossbow/actions/runs/20287538674/job/58264650270)
- [test-conda-python-3.13-pandas-nightly-numpy-nightly](https://github.com/ursacomputing/crossbow/actions/runs/20287538941/job/58264651184)
- [test-conda-python-3.13-pandas-upstream_devel-numpy-nightly](https://github.com/ursacomputing/crossbow/actions/runs/20287539903/job/58264654127)
- [test-conda-python-3.14](https://github.com/ursacomputing/crossbow/actions/runs/20287538473/job/58264649492)
### Component(s)
Python, Continuous Integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664517809
Ageed. I've consolidated all our datasets to a single bucket, such that instead of `s3://<old-dataset-bucket-name>` it's now `s3://<new-arrow-datasets-bucket>/<old-dataset-bucket-name>/`.
In this case it seems that the data should be accessible, I can open https://mf-nwp-models.s3.amazonaws.com/README.txt in my browser. Perhaps the new AWS credentials we're using are limiting access to other buckets. Let me widen the permissions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664575956
The job is green now so I am unsure if it is your change or that it was a temporary issue on their end:
https://github.com/ursacomputing/crossbow/actions/runs/20287540566/job/58297712707
We can close this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664536237
I've updated the permissions just now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664529371
Maybe it was a hiccup on that bucket, this bucket is not ours. Let me re-run some of the nightly failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664237569
We probably should use one of the ones we are providing as a project now. @rok thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok closed issue #48565: [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt
URL: https://github.com/apache/arrow/issues/48565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Unaccessible (ACCESS_DENIED) bucket used on python tests, s3://mf-nwp-models/README.txt [arrow]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #48565:
URL: https://github.com/apache/arrow/issues/48565#issuecomment-3664661466
Resolved for unknown reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org