[jira] [Commented] (ARROW-10012) [C++] Sporadic failures in CopyFiles test

["Antoine Pitrou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-10012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17196091#comment-17196091 ] 

Antoine Pitrou commented on ARROW-10012:
----------------------------------------

I checked and LocalFileSystem::CreateDir is sane (it uses what should be a race condition-immune idiom). The only hope now is to manage to reproduce locally.

A question though: why does {{CopyFiles}} create the target directories at all? Shell commands such as {{cp}} (Unix) or {{copy}} (Windows) don't do that when copying files (rather than entire directory trees). cc [~bkietz]

> [C++] Sporadic failures in CopyFiles test
> -----------------------------------------
>
>                 Key: ARROW-10012
>                 URL: https://issues.apache.org/jira/browse/ARROW-10012
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Continuous Integration
>            Reporter: Antoine Pitrou
>            Priority: Minor
>
> Here is an example failure on AppVeyor (Windows):
> https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/35210915/job/yhdxkfu7hta3kugp#L1913
> {code}
> [ RUN      ] TestSubTreeFileSystem.CopyFiles
> C:/projects/arrow/cpp/src/arrow/filesystem/filesystem_test.cc(608): error: Failed
> 'CopyFiles({{subfs_, "ab"}, {subfs_, "cd"}, {subfs_, "ef"}}, {{dest_fs, "AB/ab"}, {dest_fs, "CD/CD/cd"}, {dest_fs, "EF/EF/EF/ef"}})' failed with IOError: Path does not exist 'sub/copy/AB/ab'
> [  FAILED  ] TestSubTreeFileSystem.CopyFiles (9 ms)
> {code}
> Similar failure on Linux, showing the line number:
> https://ci.ursalabs.org/#/builders/101/builds/4412/steps/8/logs/stdio
> {code}
> [ RUN      ] TestSubTreeFileSystem.CopyFiles
> ../src/arrow/filesystem/filesystem_test.cc:608: Failure
> Failed
> 'CopyFiles({{subfs_, "ab"}, {subfs_, "cd"}, {subfs_, "ef"}}, {{dest_fs, "AB/ab"}, {dest_fs, "CD/CD/cd"}, {dest_fs, "EF/EF/EF/ef"}})' failed with IOError: Path does not exist 'sub/copy/CD/CD/cd'
> In ../src/arrow/filesystem/filesystem.cc, line 467, code: (_error_or_value13).status()
> [  FAILED  ] TestSubTreeFileSystem.CopyFiles (17 ms)
> {code}
> The erroring line number (467) corresponds to the {{OpenOutputStream}} call below:
> {code:c++}
>         auto dest_dir = internal::GetAbstractPathParent(destinations[i].path).first;
>         if (!dest_dir.empty()) {
>           RETURN_NOT_OK(destinations[i].filesystem->CreateDir(dest_dir));
>         }
> -->     ARROW_ASSIGN_OR_RAISE(
>             auto destination,
>             destinations[i].filesystem->OpenOutputStream(destinations[i].path));
> {code}
> It's not obvious why the error occurs, since the directory is created just above (race condition in the CreateDir implementation?).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)