You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/02/24 11:29:18 UTC
[jira] [Commented] (FLINK-3491) HDFSCopyUtilitiesTest fails on
Windows
[ https://issues.apache.org/jira/browse/FLINK-3491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15162778#comment-15162778 ]
ASF GitHub Bot commented on FLINK-3491:
---------------------------------------
GitHub user zentol opened a pull request:
https://github.com/apache/flink/pull/1703
[FLINK-3491] Prevent failure of HDFSCopyUtilitiesTest on Windows
This PR contains two commits to that prevent this test from failing on Windows.
The first commit resolves the problem raised in the JIRA: it changes how the URI is generated, using new Path(file).toUri() instead of file.toUri(), since the latter fails for Windows paths.
After this change, i got a new exception when running the test:
```
testCopyFromLocal(org.apache.flink.streaming.util.HDFSCopyUtilitiesTest) Time elapsed: 1.892 sec <<< ERROR!
java.lang.NullPointerException: null
at java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:739)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:722)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:631)
at org.apache.hadoop.fs.FilterFileSystem.setPermission(FilterFileSystem.java:468)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:456)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:907)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:888)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:785)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:82)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1837)
at org.apache.flink.streaming.util.HDFSCopyFromLocal$1.run(HDFSCopyFromLocal.java:49)
```
After googling a bit it appears that several hadoop versions can't deal with windows paths unless it has access to fancy libraries/dll's. (see https://issues.apache.org/jira/browse/SPARK-6961) As such I added a second commit that disables the test when run on windows.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zentol/flink 3491_test_hdfscopy
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1703.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1703
----
commit 124da4220968ce331ebd9c45cdf35e4a4074848b
Author: zentol <s....@web.de>
Date: 2016-02-24T10:14:23Z
[FLINK-3491] Prevent URIException in HDFSCopyTest
commit c458f305ad23f9f075993133ffcf5ed5dd606eb4
Author: zentol <s....@web.de>
Date: 2016-02-24T10:14:52Z
Disable HDFSCopyUtilitiesTest on Windows
----
> HDFSCopyUtilitiesTest fails on Windows
> --------------------------------------
>
> Key: FLINK-3491
> URL: https://issues.apache.org/jira/browse/FLINK-3491
> Project: Flink
> Issue Type: Bug
> Components: Tests
> Affects Versions: 1.0.0
> Reporter: Chesnay Schepler
> Assignee: Chesnay Schepler
> Fix For: 1.0.0
>
>
> testCopyFromLocal(org.apache.flink.streaming.util.HDFSCopyUtilitiesTest) Time elapsed: 0.35 sec <<< ERROR!
> java.net.URISyntaxException: Illegal character in opaque part at index 2: C:\dev\cygwin64\tmp\junit9031821586537514249\junit4281810192744805610\copy
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.<init>(URI.java:588)
> at org.apache.flink.streaming.util.HDFSCopyUtilitiesTest.testCopyFromLocal(HDFSCopyUtilitiesTest.java:57)
> testCopyToLocal(org.apache.flink.streaming.util.HDFSCopyUtilitiesTest) Time elapsed: 0.037 sec <<< ERROR!
> java.net.URISyntaxException: Illegal character in opaque part at index 2: C:\dev\cygwin64\tmp\junit6991613588668172529\junit4136927791365936888\original
> at java.net.URI$Parser.fail(URI.java:2848)
> at java.net.URI$Parser.checkChars(URI.java:3021)
> at java.net.URI$Parser.parse(URI.java:3058)
> at java.net.URI.<init>(URI.java:588)
> at org.apache.flink.streaming.util.HDFSCopyUtilitiesTest.testCopyToLocal(HDFSCopyUtilitiesTest.java:82)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)