[jira] [Comment Edited] (FLINK-4890) FileInputFormatTest#testExcludeFiles fails on Windows OS

["Chesnay Schepler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15801205#comment-15801205 ] 

Chesnay Schepler edited comment on FLINK-4890 at 1/5/17 12:12 PM:
------------------------------------------------------------------

It does kind of work.

When matching paths the we ignore schema of the path; so given an include pattern "/*/text" a path like "hdfs:/my/text" would be accepted. (since it is reduced to "/my/text" which matches the pattern)

However, if the pattern would be "hdfs:/*/text" then the matching would fail; but not just on windows but on every OS.

If we were to fix this however, and include the schema in the matching process, then it would indeed fail by default on Windows for anything but local paths.


was (Author: zentol):
It does kind of work.

When matching paths the schema of the path is ignored; so given an include pattern "/*/text" a path like "hdfs:/my/text" would be accepted. (since it is reduced to "/my/text" which matches the pattern)

However, if the pattern would be "hdfs:/*/text" then the matching would fail; but not just on windows but on every OS.

If we were to fix this however, and include the schema in the matching process, then it would indeed fail by default on Windows for anything but local paths.

> FileInputFormatTest#testExcludeFiles fails on Windows OS
> --------------------------------------------------------
>
>                 Key: FLINK-4890
>                 URL: https://issues.apache.org/jira/browse/FLINK-4890
>             Project: Flink
>          Issue Type: Bug
>          Components: Batch Connectors and Input/Output Formats
>    Affects Versions: 1.1.3
>         Environment: Windows 10
>            Reporter: Chesnay Schepler
>
> Running the mentioned test leads to an exception:
> {code}
> Illegal char <:> at index 4: file:/C:/dev/cygwin64/tmp/junit3838395086498044255/another_file.bin
> java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:/C:/dev/cygwin64/tmp/junit3838395086498044255/anot                                                                                                                     her_file.bin
>         at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
>         at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
>         at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
>         at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
>         at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
>         at java.nio.file.Paths.get(Paths.java:84)
>         at org.apache.flink.api.common.io.GlobFilePathFilter.filterPath(GlobFilePathFilter.java:95)
>         at org.apache.flink.api.common.io.FileInputFormat.acceptFile(FileInputFormat.java:644)
>         at org.apache.flink.api.common.io.FileInputFormat.addFilesInDir(FileInputFormat.java:600)
>         at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:476)
>         at org.apache.flink.api.common.io.FileInputFormatTest.testReadMultiplePatterns(FileInputFormatTest.java:362)
> {code}
> The problem is that we are given a flink Path, which is then converted to a String and given to the nio FIleSystem. The passed path is thus /C:/..., which nio can't work with.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)