You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2023/09/13 12:05:29 UTC
[GitHub] [arrow] h-vetinari opened a new issue, #37692: Compatibility with newer minio
h-vetinari opened a new issue, #37692:
URL: https://github.com/apache/arrow/issues/37692
### Describe the bug, including details regarding any error messages, version, and platform.
Conda-forge very recently added [`minio`](https://github.com/conda-forge/minio-server-feedstock), which means we could now use that, at least in the conda(-forge) jobs. I'd be interested in doing that because I see a bunch of skips in the test suite of the following kind (not exhaustive):
```
SKIPPED [2] pyarrow/tests/test_dataset.py:2623: `minio` command cannot be located
SKIPPED [1] pyarrow/tests/test_dataset.py:2638: `minio` command cannot be located
SKIPPED [1] pyarrow/tests/test_dataset.py:2664: `minio` command cannot be located
SKIPPED [1] pyarrow/tests/test_dataset.py:4754: `minio` command cannot be located
SKIPPED [1] pyarrow/tests/test_dataset.py:4821: `minio` command cannot be located
SKIPPED [1] pyarrow/tests/test_fs.py:471: `minio` command cannot be located
SKIPPED [2] pyarrow/tests/test_fs.py:562: `minio` command cannot be located
```
so if all it takes is adding the test dependency - why not?
As it turns out, this blows up pretty badly on main (through a [proxy](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1170) PR on our infra), which is at least partly, because arrow currently pins to a relatively outdated version
https://github.com/apache/arrow/blob/4fac528e2ee9c4efce453420275cbaf0b3b6adb0/ci/scripts/install_minio.sh#L54-L56
while the oldest available version in conda-forge is 2023.08.23.10.07.06.
Aside from 2-3 exceptions, the failures are all from the teardown of the `s3fs` and `py_fsspec_s3fs` fixtures, where the code unconditionally does `fs.delete_dir(bucket)`, even though the test presumably scribbled stuff in there. This leads to errors of the kind:
```
_ ERROR at teardown of test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem] _
request = <SubRequest 's3fs' for <Function test_filesystem_is_functional_after_pickling[builtin_pickle-S3FileSystem]>>
s3_server = {'connection': ('localhost', 41707, 'arrow', 'apachearrow'), 'process': <Popen: returncode: None args: ['minio', '--compat', 'server', '--quiet', '-...>, 'tempdir': local('/tmp/pytest-of-conda/pytest-0')}
@pytest.fixture
def s3fs(request, s3_server):
[...]
> fs.delete_dir(bucket)
pyarrow/tests/test_fs.py:258:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyarrow/_fs.pyx:616: in pyarrow._fs.FileSystem.delete_dir
check_status(self.fs.DeleteDir(directory))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> raise convert_status(status)
E OSError: When deleting bucket 'pyarrow-filesystem': AWS Error UNKNOWN (HTTP status 409) during DeleteBucket operation: Unable to parse ExceptionName: BucketNotEmpty Message: The bucket you tried to delete is not empty
pyarrow/error.pxi:91: OSError
```
more specifically, close to ~60 of them:
```
= 4 failed, 7276 passed, 223 skipped, 19 deselected, 28 xfailed, 2 xpassed, 8 warnings, 57 errors in 177.84s (0:02:57) =
```
My first guess would be that:
* either minio got stricter (less buggier) in when it allows deletion of non-empty buckets
* or something in the conda-forge setup does not yet accurately reproduce what the CI is doing here.
I'm quite out of my depths here, but I think a alternative would be to somehow pipe through [`ForceBucketDelete`](https://min.io/docs/minio/linux/administration/identity-access-management/policy-based-access-control.html#policy-action.s3-ForceDeleteBucket) or `--force`. A cheap alternative is the following patch (also doesn't cover 100%), which just doesn't care about failed bucket deletes:
```diff
--- a/python/pyarrow/tests/test_fs.py
+++ b/python/pyarrow/tests/test_fs.py
@@ -256,7 +256,10 @@ def s3fs(request, s3_server):
allow_move_dir=False,
allow_append_to_file=False,
)
- fs.delete_dir(bucket)
+ try:
+ fs.delete_dir(bucket)
+ except OSError:
+ pass
@pytest.fixture
@@ -358,7 +361,10 @@ def py_fsspec_s3fs(request, s3_server):
allow_move_dir=False,
allow_append_to_file=True,
)
- fs.delete_dir(bucket)
+ try:
+ fs.delete_dir(bucket)
+ except OSError:
+ pass
@pytest.fixture(params=[
```
After that patch, the only remaining errors are then:
```
FAILED pyarrow/tests/test_fs.py::test_get_file_info[PyFileSystem(FSSpecHandler(s3fs.S3FileSystem()))] - AssertionError: assert <FileType.Directory: 3> == <FileType.NotFound: 0>
+ where <FileType.Directory: 3> = <FileInfo for 'pyarrow-filesystem/a/aa/aaa/': type=FileType.Directory>.type
+ and <FileType.NotFound: 0> = FileType.NotFound
FAILED pyarrow/tests/test_fs.py::test_delete_dir[S3FileSystem] - Failed: DID NOT RAISE <class 'OSError'>
FAILED pyarrow/tests/test_fs.py::test_delete_dir_contents[S3FileSystem] - Failed: DID NOT RAISE <class 'OSError'>
FAILED pyarrow/tests/test_fs.py::test_move_directory[PyFileSystem(FSSpecHandler(s3fs.S3FileSystem()))] - Failed: DID NOT RAISE <class 'OSError'>
```
# This issue does not happen on 13.0.0?!
The final kicker is that _**all this is passing with arrow 13**_ - I even checked that the tests didn't get skipped. So it appears there are at least two things at work here: a change in minio behaviour & a change in pyarrow somewhere.
### Component(s)
Continuous Integration, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #37692: [Python][CI] Compatibility with newer minio
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-1729169675
@h-vetinari Can you try running the Arrow C++ tests with this new Minio version and report the results?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #37692: [Python][CI] Compatibility with newer minio
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-1729236283
Wait, no, the CI run you're pointing to is showing the PyArrow tests. I'm talking about the Arrow C++ tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #37692: [Python][CI] Compatibility with newer minio
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-1729217695
I don't understand - that's exactly what I did in https://github.com/conda-forge/arrow-cpp-feedstock/pull/1170. You can look at one of the resulting CI runs [here](https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=779614&view=results)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Compatibility with newer minio [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-2371119035
@amoeba Thanks for the report. It should be useful when someone tries to update Minio on CI, I guess.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #37692: [Python][CI] Compatibility with newer minio
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-1729233934
Ok, but if you don't report the results on the Arrow issue tracker, then nobody notices :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Compatibility with newer minio [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-2418402586
For the purposes of testing the python-bits, this issue has been solved in 18.0.0rc0 (presumably by https://github.com/apache/arrow/pull/44225) - thank you! I've been waiting for arrow v18 to enable testing the C++ library as well, will report back in #35587.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Compatibility with newer minio [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari closed issue #37692: [Python][CI] Compatibility with newer minio
URL: https://github.com/apache/arrow/issues/37692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python][CI] Compatibility with newer minio [arrow]
Posted by "amoeba (via GitHub)" <gi...@apache.org>.
amoeba commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-2369933905
@pitrou in case it's helpful, I ran the test suite from a recent C++ build with minio `RELEASE.2024-09-09T16-59-28Z` on macOS aarch64 and got a few failures. Here's one as an example:
```
[ RUN ] TestS3FS.GetFileInfoSelectorRecursive
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/s3fs_test.cc:826: Failure
Failed
'_error_or_value66.status()' failed with IOError: When listing objects under key '' in bucket 'empty-bucket': AWS Error UNKNOWN (HTTP status 429) during ListObjectsV2 operation: Unable to parse ExceptionName: TooManyRequests Message: Please reduce your request rate
```
Key part there being "Please reduce your request rate".
I then set `export MINIO_API_REQUESTS_MAX=1600` and then only got two failures which should be easily resolved. It looks like some behavior changed in minio:
```
[ RUN ] TestS3FS.CreateDir
/Users/bryce/src/apache/arrow/cpp/src/arrow/filesystem/s3fs_test.cc:1011: Failure
Failed
Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #37692: [Python][CI] Compatibility with newer minio
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #37692:
URL: https://github.com/apache/arrow/issues/37692#issuecomment-1729254345
> I'm talking about the Arrow C++ tests.
Ah, sorry for the misunderstanding. I've been trying to enable the C++ tests in https://github.com/conda-forge/arrow-cpp-feedstock/pull/1058, but it's not passing yet (completely aside from this issue here).
> Ok, but if you don't report the results on the Arrow issue tracker, then nobody notices :-)
I had opened https://github.com/apache/arrow/issues/35587 for that. 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org