You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Zhe Yu (Jira)" <ji...@apache.org> on 2020/05/03 01:49:00 UTC
[jira] [Comment Edited] (FLINK-16198) FileUtilsTest fails on Mac OS
[ https://issues.apache.org/jira/browse/FLINK-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098164#comment-17098164 ]
Zhe Yu edited comment on FLINK-16198 at 5/3/20, 1:48 AM:
---------------------------------------------------------
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing file for testing.
*Solution*: Since goal of this test case is to verify correctness of FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute path while we will still use relative path to test FileUtils method will be sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from root directory: a. Thus, race condition will happen. There will be a chance 2 threads trying to delete /a/b/e/ at the same time. When this happens the test will file. Maybe I'm missing something given [this comment|#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|#L317]] synchronized will protect the test from failing. But it also means concurrent deletion is disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to me. I'm also more than glad to talk about both of them.
was (Author: fishzhe):
h3. Failed test case 1: FileUtilsTest.testCompressionOnRelativePath
*Root cause*: failed to create directory using relative path when preparing file for testing.
*Solution*: Since goal of this test case is to verify correctness of FileUtils.compressDirectory and FileUtils.expandDirectory
Thus refactor verifyDirectoryCompression to prepare test data using absolute path while we will still use relative path to test FileUtils method will be sufficient.
h3. Failed test case 2: FileUtilsTest.testDeleteDirectoryConcurrently
*Root cause:* FileUtils.deleteDirectory is not thread safe. This test case is not always failing. In my local env(MacOS), it failed 7,8 times out of 10. The test case is creating 3 level file hierarchy like below:
- a
- b
- e
- c
- d
Then the test case started 3 threads. All of them are trying to iterate from root directory: a. Thus, race condition will happen. There will be a chance 2 threads trying to delete /a/b/e/ at the same time. When this happens the test will file. Maybe I'm missing something given [this comment|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L248]]
*Solution:* make [deleteFileOrDirectoryInternal|[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/util/FileUtils.java#L317]] synchronized will protect the test from failing. But it also means concurrent deletion is disabled if 2 threads trying to delete from the same root.
If above proposal sounds reasonable to you, feel free to assign this issue to me. I'm also more than glad to talk about both of them.
> FileUtilsTest fails on Mac OS
> -----------------------------
>
> Key: FLINK-16198
> URL: https://issues.apache.org/jira/browse/FLINK-16198
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
> Affects Versions: 1.11.0
> Reporter: Andrey Zagrebin
> Priority: Blocker
> Labels: starter
> Fix For: 1.11.0
>
>
> The following tests fail if run on Mac OS (IDE/maven).
>
> FileUtilsTest.testCompressionOnRelativePath:
> {code:java}
> java.nio.file.NoSuchFileException: ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit6496651678375117676/compressDir/rootDirjava.nio.file.NoSuchFileException: ../../../../../var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit6496651678375117676/compressDir/rootDir
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) at java.nio.file.Files.createDirectory(Files.java:674) at org.apache.flink.util.FileUtilsTest.verifyDirectoryCompression(FileUtilsTest.java:440) at org.apache.flink.util.FileUtilsTest.testCompressionOnRelativePath(FileUtilsTest.java:261) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.junit.runner.JUnitCore.run(JUnitCore.java:137) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {code}
>
> FileUtilsTest.testDeleteDirectoryConcurrently:
> {code:java}
> java.nio.file.FileSystemException: /var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66: Invalid argument at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:276)
> at org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:515)
> at org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)java.nio.file.FileSystemException: /var/folders/67/v4yp_42d21j6_n8k1h556h0c0000gn/T/junit7558825557740784886/junit3566161583262218465/ab1fa0bde8b22cad58b717508c7a7300/121fdf5f7b057183843ed2e1298f9b66/6598025f390d3084d69c98b36e542fe2/8db7cd9c063396a19a86f5b63ce53f66: Invalid argument
> at sun.nio.fs.UnixException.translateToIOException(UnixException.java:91)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244)
> at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
> at java.nio.file.Files.deleteIfExists(Files.java:1165)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:324)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectoryInternal(FileUtils.java:320)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteFileOrDirectory(FileUtils.java:258)
> at org.apache.flink.util.FileUtils.cleanDirectoryInternal(FileUtils.java:376)
> at org.apache.flink.util.FileUtils.deleteDirectoryInternal(FileUtils.java:335)
> at org.apache.flink.util.FileUtils.guardIfWindows(FileUtils.java:391)
> at org.apache.flink.util.FileUtils.deleteDirectory(FileUtils.java:276)
> at org.apache.flink.util.FileUtilsTest$Deleter.go(FileUtilsTest.java:515)
> at org.apache.flink.core.testutils.CheckedThread.run(CheckedThread.java:74)%MCEPASTEBIN%{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)