You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "llama90 (via GitHub)" <gi...@apache.org> on 2024/07/14 06:43:56 UTC
[I] [C++][FileSystem] Is it true that a File and a Directory cannot have the Same name? [arrow]
llama90 opened a new issue, #43247:
URL: https://github.com/apache/arrow/issues/43247
### Describe the usage question you have. Please include as many useful details as possible.
I've updated Minio to the latest version for the "[CI] Update minio version(#41922)" issue. While reviewing the failing tests, I found something unclear and would like to get opinion on it.
In both file systems and object storage, a file (object) can have the same name as a directory (prefix). However, there's a test that expects directory creation to fail if a file with the same name already exists.
**Test Initialization**
https://github.com/apache/arrow/blob/718415030f8ebcad14bc3fa10906beca6526c5e9/cpp/src/arrow/filesystem/s3fs_test.cc#L464-L466
During initialization, a file is created, but the code tests for the failure of directory creation.
https://github.com/apache/arrow/blob/718415030f8ebcad14bc3fa10906beca6526c5e9/cpp/src/arrow/filesystem/s3fs_test.cc#L954-L955
Should we continue to follow this rule that files and directories can't have the same name?
This test was created a long time ago (August 2019) and Minio is running on version `2022.05-26T05-48-41Z`, so it might not be up to date.
- https://github.com/apache/arrow/commit/7ec173172953f6ab27f7bad495bd49bff451d03c
If it's correct that files and directories can have the same name, the test should be updated.
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][FileSystem] Is it true that a File and a Directory cannot have the Same name? [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #43247:
URL: https://github.com/apache/arrow/issues/43247#issuecomment-2227380115
When upgrading the MinIO version, similar errors occur, leading to test failures:
<details><summary>arrow-s3fs-test result</summary>
```
Testing started at 오전 12:00 ...
UpdateCTestConfiguration from :/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/DartConfiguration.tcl
UpdateCTestConfiguration from :/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/DartConfiguration.tcl
Test project /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
70: Test command: /Users/lama/workspace/arrow-new/cpp/build-support/run-test.sh "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems" "test" "/Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/debug//arrow-s3fs-test"
70: Working Directory: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
70: Test timeout computed to be: 10000000
70: Running arrow-s3fs-test, redirecting output into /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/build/test-logs/arrow-s3fs-test.txt (attempt 1/1)
70: Running main() from /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/_deps/googletest-src/googletest/src/gtest_main.cc
70: [==========] Running 69 tests from 7 test suites.
70: [----------] Global test environment set-up.
70: [----------] 3 tests from S3OptionsTest
70: [ RUN ] S3OptionsTest.FromUri
70: [ OK ] S3OptionsTest.FromUri (114 ms)
70: [ RUN ] S3OptionsTest.FromAccessKey
70: [ OK ] S3OptionsTest.FromAccessKey (0 ms)
70: [ RUN ] S3OptionsTest.FromAssumeRole
70: [ OK ] S3OptionsTest.FromAssumeRole (1 ms)
70: [----------] 3 tests from S3OptionsTest (115 ms total)
70:
70: [----------] 4 tests from S3RegionResolutionTest
70: [ RUN ] S3RegionResolutionTest.PublicBucket
70: [ OK ] S3RegionResolutionTest.PublicBucket (34 ms)
70: [ RUN ] S3RegionResolutionTest.RestrictedBucket
70: [ OK ] S3RegionResolutionTest.RestrictedBucket (32 ms)
70: [ RUN ] S3RegionResolutionTest.NonExistentBucket
70: [ OK ] S3RegionResolutionTest.NonExistentBucket (1244 ms)
70: [ RUN ] S3RegionResolutionTest.InvalidBucketName
70: [ OK ] S3RegionResolutionTest.InvalidBucketName (0 ms)
70: [----------] 4 tests from S3RegionResolutionTest (1311 ms total)
70:
70: [----------] 2 tests from S3FileSystemRegionTest
70: [ RUN ] S3FileSystemRegionTest.Default
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:397: Failure
70: Expected equality of these values:
70: s3fs->region()
70: Which is: "ap-northeast-2"
70: "us-east-1"
70: [ FAILED ] S3FileSystemRegionTest.Default (0 ms)
70: [ RUN ] S3FileSystemRegionTest.EnvironmentVariable
70: [ OK ] S3FileSystemRegionTest.EnvironmentVariable (2 ms)
70: [----------] 2 tests from S3FileSystemRegionTest (2 ms total)
70:
70: [----------] 1 test from TestMinioServer
70: [ RUN ] TestMinioServer.Connect
70: [ OK ] TestMinioServer.Connect (928 ms)
70: [----------] 1 test from TestMinioServer (928 ms total)
70:
70: [----------] 32 tests from TestS3FS
70: [ RUN ] TestS3FS.GetFileInfoRoot
70: [ OK ] TestS3FS.GetFileInfoRoot (190 ms)
70: [ RUN ] TestS3FS.GetFileInfoBucket
70: [ OK ] TestS3FS.GetFileInfoBucket (93 ms)
70: [ RUN ] TestS3FS.GetFileInfoObject
70: [ OK ] TestS3FS.GetFileInfoObject (126 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelector
70: [ OK ] TestS3FS.GetFileInfoSelector (192 ms)
70: [ RUN ] TestS3FS.GetFileInfoSelectorRecursive
70: [ OK ] TestS3FS.GetFileInfoSelectorRecursive (182 ms)
70: [ RUN ] TestS3FS.GetFileInfoGenerator
70: [ OK ] TestS3FS.GetFileInfoGenerator (103 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorStress
70: [ OK ] TestS3FS.GetFileInfoGeneratorStress (7857 ms)
70: [ RUN ] TestS3FS.GetFileInfoGeneratorCancelled
70: [ OK ] TestS3FS.GetFileInfoGeneratorCancelled (63 ms)
70: [ RUN ] TestS3FS.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/s3fs_test.cc:967: Failure
70: Failed
70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FS.CreateDir (167 ms)
70: [ RUN ] TestS3FS.DeleteFile
70: [ OK ] TestS3FS.DeleteFile (151 ms)
70: [ RUN ] TestS3FS.DeleteDir
70: [ OK ] TestS3FS.DeleteDir (294 ms)
70: [ RUN ] TestS3FS.DeleteDirContents
70: [ OK ] TestS3FS.DeleteDirContents (261 ms)
70: [ RUN ] TestS3FS.DeleteDirContentsAsync
70: [ OK ] TestS3FS.DeleteDirContentsAsync (211 ms)
70: [ RUN ] TestS3FS.CopyFile
70: [ OK ] TestS3FS.CopyFile (197 ms)
70: [ RUN ] TestS3FS.Move
70: [ OK ] TestS3FS.Move (213 ms)
70: [ RUN ] TestS3FS.OpenInputStream
70: [ OK ] TestS3FS.OpenInputStream (118 ms)
70: [ RUN ] TestS3FS.OpenInputStreamMetadata
70: [ OK ] TestS3FS.OpenInputStreamMetadata (95 ms)
70: [ RUN ] TestS3FS.OpenInputFile
70: [ OK ] TestS3FS.OpenInputFile (134 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamBackgroundWrites (954 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamSyncWrites (733 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortBackgroundWrites (90 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAbortSyncWrites
70: [ OK ] TestS3FS.OpenOutputStreamAbortSyncWrites (122 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamDestructorBackgroundWrites (146 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamDestructorSyncWrite (146 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorBackgroundWrites (161 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamAsyncDestructorSyncWrite (174 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockBackgroundWrites (169 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite
70: [ OK ] TestS3FS.OpenOutputStreamCloseAsyncFutureDeadlockSyncWrite (227 ms)
70: [ RUN ] TestS3FS.OpenOutputStreamMetadata
70: [ OK ] TestS3FS.OpenOutputStreamMetadata (245 ms)
70: [ RUN ] TestS3FS.FileSystemFromUri
70: [ OK ] TestS3FS.FileSystemFromUri (88 ms)
70: [ RUN ] TestS3FS.NoCreateDeleteBucket
70: [ OK ] TestS3FS.NoCreateDeleteBucket (117 ms)
70: [ RUN ] TestS3FS.CustomRetryStrategy
70: [ OK ] TestS3FS.CustomRetryStrategy (105 ms)
70: [----------] 32 tests from TestS3FS (14141 ms total)
70:
70: [----------] 26 tests from TestS3FSGeneric
70: [ RUN ] TestS3FSGeneric.Empty
70: [ OK ] TestS3FSGeneric.Empty (76 ms)
70: [ RUN ] TestS3FSGeneric.NormalizePath
70: [ OK ] TestS3FSGeneric.NormalizePath (54 ms)
70: [ RUN ] TestS3FSGeneric.CreateDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:244: Failure
70: Failed
70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CreateDir (255 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDir
70: [ OK ] TestS3FSGeneric.DeleteDir (310 ms)
70: [ RUN ] TestS3FSGeneric.DeleteDirContents
70: [ OK ] TestS3FSGeneric.DeleteDirContents (298 ms)
70: [ RUN ] TestS3FSGeneric.DeleteRootDirContents
70: [ OK ] TestS3FSGeneric.DeleteRootDirContents (179 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFile
70: [ OK ] TestS3FSGeneric.DeleteFile (297 ms)
70: [ RUN ] TestS3FSGeneric.DeleteFiles
70: [ OK ] TestS3FSGeneric.DeleteFiles (437 ms)
70: [ RUN ] TestS3FSGeneric.MoveFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:450: Failure
70: Failed
70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.MoveFile (552 ms)
70: [ RUN ] TestS3FSGeneric.MoveDir
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:461: Skipped
70: Filesystem doesn't allow moving directories
70: [ SKIPPED ] TestS3FSGeneric.MoveDir (53 ms)
70: [ RUN ] TestS3FSGeneric.CopyFile
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:572: Failure
70: Failed
70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
70: [ FAILED ] TestS3FSGeneric.CopyFile (390 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfo
70: [ OK ] TestS3FSGeneric.GetFileInfo (211 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoVector
70: [ OK ] TestS3FSGeneric.GetFileInfoVector (222 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelector
70: [ OK ] TestS3FSGeneric.GetFileInfoSelector (202 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion
70: [ OK ] TestS3FSGeneric.GetFileInfoSelectorWithRecursion (236 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoAsync
70: [ OK ] TestS3FSGeneric.GetFileInfoAsync (139 ms)
70: [ RUN ] TestS3FSGeneric.GetFileInfoGenerator
70: [ OK ] TestS3FSGeneric.GetFileInfoGenerator (216 ms)
70: [ RUN ] TestS3FSGeneric.OpenOutputStream
70: [ OK ] TestS3FSGeneric.OpenOutputStream (279 ms)
70: [ RUN ] TestS3FSGeneric.OpenAppendStream
70: /Users/lama/workspace/arrow-new/cpp/src/arrow/filesystem/test_util.cc:946: Skipped
70: Filesystem doesn't allow file appends
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream (48 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStream
70: [ OK ] TestS3FSGeneric.OpenInputStream (189 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputStreamWithFileInfo (190 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputStreamAsync
70: [ OK ] TestS3FSGeneric.OpenInputStreamAsync (157 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFile
70: [ OK ] TestS3FSGeneric.OpenInputFile (184 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileWithFileInfo
70: [ OK ] TestS3FSGeneric.OpenInputFileWithFileInfo (204 ms)
70: [ RUN ] TestS3FSGeneric.OpenInputFileAsync
70: [ OK ] TestS3FSGeneric.OpenInputFileAsync (142 ms)
70: [ RUN ] TestS3FSGeneric.SpecialChars
70: [ OK ] TestS3FSGeneric.SpecialChars (284 ms)
70: [----------] 26 tests from TestS3FSGeneric (5821 ms total)
70:
70: [----------] 1 test from S3GlobalOptions
70: [ RUN ] S3GlobalOptions.DefaultsLogLevel
70: [ OK ] S3GlobalOptions.DefaultsLogLevel (0 ms)
70: [----------] 1 test from S3GlobalOptions (0 ms total)
70:
70: [----------] Global test environment tear-down
70: [==========] 69 tests from 7 test suites ran. (22363 ms total)
70: [ PASSED ] 62 tests.
70: [ SKIPPED ] 2 tests, listed below:
70: [ SKIPPED ] TestS3FSGeneric.MoveDir
70: [ SKIPPED ] TestS3FSGeneric.OpenAppendStream
70: [ FAILED ] 5 tests, listed below:
70: [ FAILED ] S3FileSystemRegionTest.Default
70: [ FAILED ] TestS3FS.CreateDir
70: [ FAILED ] TestS3FSGeneric.CreateDir
70: [ FAILED ] TestS3FSGeneric.MoveFile
70: [ FAILED ] TestS3FSGeneric.CopyFile
70:
70: 5 FAILED TESTS
70: ~/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/src/arrow/filesystem
Failed
0% tests passed, 1 tests failed out of 1
Label Time Summary:
arrow-tests = 25.25 sec*proc (1 test)
filesystem = 25.25 sec*proc (1 test)
unittest = 25.25 sec*proc (1 test)
Total Test time (real) = 25.26 sec
The following tests FAILED:
70 - arrow-s3fs-test (Failed)
Errors while running CTest
Output from these tests are in: /Users/lama/workspace/arrow-new/cpp/cmake-build-debug-preset-filesystems/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
Process finished with exit code 8
```
</details>
- 70: Expected 'fs_->CreateDir("bucket/somefile")' to fail with IOError, but got OK
- 70: Expected 'fs->CreateDir("AB/def/EF/GH", true )' to fail with IOError, but got OK
- 70: Expected 'fs->Move("AB/pqr", "xxx/mno")' to fail with IOError, but got OK
- 70: Expected 'fs->CopyFile("AB/abc", "def/mno")' to fail with IOError, but got OK
Although the MinIO release notes need to be reviewed in more detail, it appears that MinIO might not be checking whether the file or directory already exists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][FileSystem] Is it true that a File and a Directory cannot have the Same name? [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 commented on issue #43247:
URL: https://github.com/apache/arrow/issues/43247#issuecomment-2227510974
For precise discussion, I need to understand what changes were made in MinIO. Therefore, I am closing this issue for now.
- minio.RELEASE.2022-05-26T05-48-41Z
- https://github.com/minio/minio/compare/RELEASE.2024-07-10T18-41-49Z...RELEASE.2024-07-13T01-46-15Z
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][FileSystem] Is it true that a File and a Directory cannot have the Same name? [arrow]
Posted by "llama90 (via GitHub)" <gi...@apache.org>.
llama90 closed issue #43247: [C++][FileSystem] Is it true that a File and a Directory cannot have the Same name?
URL: https://github.com/apache/arrow/issues/43247
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org