You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "lohit vijayarenu (JIRA)" <ji...@apache.org> on 2007/10/16 23:52:50 UTC
[jira] Created: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
filenames with ':' colon throws java.lang.IllegalArgumentException
------------------------------------------------------------------
Key: HADOOP-2066
URL: https://issues.apache.org/jira/browse/HADOOP-2066
Project: Hadoop
Issue Type: Bug
Components: dfs
Reporter: lohit vijayarenu
File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
[lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
URI: testfile-2007-09-24-03:00:00.gz
at org.apache.hadoop.fs.Path.initialize(Path.java:140)
at org.apache.hadoop.fs.Path.<init>(Path.java:126)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.<init>(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
... 10 more
[lohit@krygw1000 ~]$
Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2066) filenames with ':' colon
throws java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535983 ]
szetszwo edited comment on HADOOP-2066 at 10/18/07 10:37 AM:
---------------------------------------------------------------------------
> How do we know what parts are the filename?
Could we assume the URIs are hierarchical URIs (defined in the [java.net.URI|http://java.sun.com/javase/6/docs/api/java/net/URI.html])? If yes and we further force the rule that scheme must be specified with authority,
then the syntax will be
[[scheme:]//authority]path
I think the colons can be distinguished in this case.
I agree that the current patch is simple and clean. However, users cannot perform something like
{code}
> ls
a:b foo
> bin/hadoop fs -put a* baz
{code}
was (Author: szetszwo):
> How do we know what parts are the filename?
Could we assume the URIs are hierarchical URIs (defined in the [java.net.URI|http://java.sun.com/javase/6/docs/api/java/net/URI.html])? If yes and we further force the rule that scheme must be specified with authority,
then the syntax will be
[[scheme:]//authority]path
I think the colons can be distinguished in this case.
I agree that the current patch is simple and clean. However, users cannot perform something like
{code}
> ls
a:b foo
> bin/hadoop fs -put * baz
{code}
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535378 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
The bug is not that this fails, but that it still fails even if you escape the colons, turning them into '%3a'. Path(String) does not properly handle escapes.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535647 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
By, "fixes this", I mean this permits one to include colons in filenames by using the escape sequence "%3a". In particular, 'bin/hadoop fs -put foo%3abar baz' will copy the local file named 'foo:bar'.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558928#action_12558928 ]
eric baldeschwieler commented on HADOOP-2066:
---------------------------------------------
Good idea. Something like this would address my concerns.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535695 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
> Do you want to use java.net.URLEncoder to encode filename, so that 'bin/hadoop fs -put foo:bar baz' will work?
That would break things like 'bin/hadoop fs -cp hdfs://a:999/foo/bar baz'. Colons have an interpretation in paths, that's why they need to be escaped.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536836 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
I found that the URI constructor indeed can handle special character like ':'. The error reported by Lohit is another issue.
The first URI below will success but the second one will fail.
{code}
new URI("http", "www.apache.com", "/a:b", null, null);
new URI("http", "www.apache.com", "./a:b", null, null);
{code}
That's why the error message says "Relative path in absolute URI". I will think about how to fix it.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535683 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
Do you want to use java.net.URLEncoder to encode filename, so that 'bin/hadoop fs -put foo:bar baz' will work?
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536758 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
Nicholas: it looks like you've only addressed including colons within "file:" URIs. Is there a reason for this? That seems rather arbitrary to me. I would much prefer a general solution (like the one I attached).
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535497 ]
eric baldeschwieler commented on HADOOP-2066:
---------------------------------------------
The API used does not look like a URI API, hence a URI quoting requirement also seems wrong.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
lohit vijayarenu updated HADOOP-2066:
-------------------------------------
Description:
File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
URI: testfile-2007-09-24-03:00:00.gz
at org.apache.hadoop.fs.Path.initialize(Path.java:140)
at org.apache.hadoop.fs.Path.<init>(Path.java:126)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.<init>(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
... 10 more
Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
scheme, which in this case is clearly not a valid scheme.
was:
File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
[lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
URI: testfile-2007-09-24-03:00:00.gz
at org.apache.hadoop.fs.Path.initialize(Path.java:140)
at org.apache.hadoop.fs.Path.<init>(Path.java:126)
at org.apache.hadoop.fs.Path.<init>(Path.java:50)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
at java.net.URI.checkPath(URI.java:1787)
at java.net.URI.<init>(URI.java:735)
at org.apache.hadoop.fs.Path.initialize(Path.java:137)
... 10 more
[lohit@krygw1000 ~]$
Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
scheme, which in this case is clearly not a valid scheme.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535994 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
Nicholas, yes, perhaps we can optimize our parsing of URIs to accept more cases, so that, in particular, colons can be included in relative paths, excluding only relative paths that contain '://'. Should we still support escapes? Would you like to make a patch for this? Passing tests on Windows will be the hard part. I've not yet tested my patch on Windows and assume it will fail. It would be easier if we don't support escapes...
Lohit, HDFS does not permit colons in file names, while Unix does. This is hard-coded into HDFS. It could perhaps be changed. It was done to err on the safe side: we did not want to end up with files whose names were illegal, and hence might, e.g., not be removed or read, and chose to check for characters which could prove problematic and prohibit them.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535371 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
I guess / is the only invalid character for the path in Linux. All filenames below are valid.
{code}
A#B A$B A%B A&B A*B A+B A,B A:B A;B A=B A?B A[B A\B A^B A{B
{code}
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537073 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
> users expect to be able to specify posix like file strings
Which users? Not Windows or MacOS X users, since those OSes don't have POSIX file names, especially when it comes to colons. Linux users may expect posix names, but Hadoop is cross-platform.
URIs are a good standard for our pathnames, a better standard than POSIX. URIs can handle platform-dependent characters like colons just fine. We have in the past made some exceptions in URI syntax to simplify platform compatibility. In particular, we interpret "C:\foo\bar" as "file:///c:/foo/bar" without requiring escapes. Perhaps we should continue this trend further, as Nicholas has proposed, perhaps any exceptions are a mistake, or perhaps we should not make any new exceptions. That seems like a reasonable discussion. But I don't see the case to abandon URIs entirely as our primary filename syntax specification. Handling non-portable filename characters like colons with or without quotation seems a minor issue next to having an i18n-safe standard that extensibly handles protocols, hosts, ports, etc. I don't consider that unwanted baggage.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536750 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
Hi Lohit,
The message says, "abcd\:efgh.tar.gz is not a valid DFS filename." because HDFS does not permit special characters like colon (see FSNamesystem.isValidName). This is not a parsing problem. If you try something like
{code}hadoop dfs -put ./a:b ./ab{code}
i.e. put and rename the file. It should work.
Nicholas
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536781 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
I intend to force the rule "scheme, if there is any, must be specified with authority." However, there are quite a few places in the current codes that using path uri like "file:/foo/bar". If we don't allow "file:" without authority, we might have a compatibility issue.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2066:
-------------------------------------------
Attachment: 2066_20071022.patch
In 2066_20071022.patch, we handle colons as following
- If the input URI is absolute, then the colons before the root slash are for scheme or authority and the colons after it are for path.
- For relative URIs, if colons occur after a slash, they must be a part of file/directory names. No problem.
- If the URI is relative and the first colon occur before any slash, then the colon will be misinterpreted for scheme in the URI constructor. Therefore, {{./}} is inserted in the beginning in {{Path.normalizePath}}.
- For some wired relative URI like {{hdfs://apache.com/foo}} (i.e. {{hdfs:}} and {{apache.com}} are directories), then users have to add {{./}} themselves (i.e. {{./hdfs://apache.com/foo}})
Another question: should we allow colons in HDFS?
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558464#action_12558464 ]
eric baldeschwieler commented on HADOOP-2066:
---------------------------------------------
I think it makes great sense to support a URI interface to the FS. I recall championing that.
I'm not at all sure it makes sense to define what is a valid filename based on a URI library.
URI can support quoting and we will undoubtedly support several non-URI interfaces to the FS.
I can create filenames with colon's on my mac btw. Not that I see why our filenames should be defined by the limits of what all supported client platforms can support.
I think this issue requires more thought.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-2066:
---------------------------------
Attachment: HADOOP-2066.patch
> The API used does not look like a URI API, hence a URI quoting requirement also seems wrong.
It is like a URI constructor. URI cannot be subclassed, and we want to add behavior, so we wrap our URIs in Paths. The javadoc for the Path constructor says, and has always said, "Path strings are URIs". We long-ago settled on URI syntax for file paths. So URI escapes seem appropriate.
I've attached a patch which fixes this on Linux. Who knows what havoc it will wreak on Windows, though, where colons in paths are more common, and requiring applications to escape them, while correct, may prove painful.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12558701#action_12558701 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
> I'm not at all sure it makes sense to define what is a valid filename based on a URI library.
The URI standard provides a good interchange syntax for file names. But we shouldn't let it limit what names are possible in various filesystems as we do today: we should support their full range, using escapes where necessary. Unfortunately, with the current API, we can't tell when a character needs to be escaped or when it is intended as a URI meta-character.
The problem is that we construct paths in FileSystem independent code, so we don't know how to escape things. Perhaps the solution is to remove the public Path constructor and force all Paths to be created by a FileSystem#createPath method, so that they can be escaped appropriately.
Thus, when running on Windows, if one passes a string with unescaped backslashes to LocalFileSystem#createPath(), the backslashes would be interpreted as directory separators, while on Linux or HDFS they'd be treated as literals. Unescaped slashes in a Path URI will always be directory separators, since that's the URI standard we're using for interchange.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535736 ]
Doug Cutting commented on HADOOP-2066:
--------------------------------------
> We could only encode filenames and directory names but not the entire string. Then, all characters except / can be allowed.
How do we know what parts are the filename? The colon separates the scheme from the rest of the uri, so we cannot know whether a colon in a name delimits the scheme or part of the name.
> Also, we probably should provide a option to allow user giving a raw URI.
We only support a subset of URIs, so we need to have our own constructors to validate and normalize. We could accept a URI and check that it is of the form we require. The easiest implementation would be to convert it to a string and pass it to the existing constructor. What's the use case?
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2066:
-------------------------------------------
Attachment: 2066_20071019.patch
- All characters except slash are valid for directory/file names
- Syntax: {code}[[<scheme>:]//<authority>]<path> | file:<path>{code}
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-2066) filenames with ':' colon
throws java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535371 ]
szetszwo edited comment on HADOOP-2066 at 10/16/07 4:47 PM:
--------------------------------------------------------------------------
I guess / is the only invalid character for the filename in Linux (GNU/Linux). All filenames below are valid.
{code}
A#B A$B A%B A&B A*B A+B A,B A:B A;B A=B A?B A[B A\B A^B A{B
{code}
was (Author: szetszwo):
I guess / is the only invalid character for the path in Linux. All filenames below are valid.
{code}
A#B A$B A%B A&B A*B A+B A,B A:B A;B A=B A?B A[B A\B A^B A{B
{code}
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535701 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
We could only encode filenames and directory names but not the entire string. Then, all characters except / can be allowed.
Also, we probably should provide a option to allow user giving a raw URI.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> [lohit@krygw1000 ~]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> [lohit@krygw1000 ~]$
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536001 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
Sure, I will make a patch.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535986 ]
lohit vijayarenu commented on HADOOP-2066:
------------------------------------------
How would be the usage with this patch?
I tried these
[lohit@hadoop-trunk]$ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz "testfile-2007-09-24-03%3a00%3a00.gz"
put: Pathname /user/lohit/testfile-2007-09-24-03:00:00.gz from testfile-2007-09-24-03:00:00.gz is not a valid DFS filename.
[lohit@hadoop-trunk]$ echo "TEST" > testfile
[lohit@hadoop-trunk]$ hadoop dfs -put ./testfile "testfile-2007-09-24-03%3a00%3a00.gz"
put: Pathname /user/lohit/testfile-2007-09-24-03:00:00.gz from testfile-2007-09-24-03:00:00.gz is not a valid DFS filename.
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "lohit vijayarenu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536716 ]
lohit vijayarenu commented on HADOOP-2066:
------------------------------------------
Hi Nicholas,
I just tried this patch against trunk.
Am I doing this right?
[lohit@ hadoop-trunk]$ hadoop dfs -put ./abcd:efgh.tar.gz "abcd\:efgh.tar.gz"
put: Pathname /user/lohit/abcd\:efgh.tar.gz from abcd\:efgh.tar.gz is not a valid DFS filename.
[lohit@ hadoop-trunk]$ hadoop dfs -put ./abcd:efgh.tar.gz "abcd%3aefgh.tar.gz"
[lohit@ hadoop-trunk]$ hadoop dfs -ls
Found 3 items
/user/lohit/abcd <r 3> 5 2007-10-22 15:38
/user/lohit/abcd%3aefgh.tar.gz <r 3> 5 2007-10-22 15:39
/user/lohit/test.x <r 3> 61646 2007-10-22 15:38
[lohit@ucdev27 hadoop-trunk]$
Thanks!
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071019.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12535983 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-2066:
------------------------------------------------
> How do we know what parts are the filename?
Could we assume the URIs are hierarchical URIs (defined in the [java.net.URI|http://java.sun.com/javase/6/docs/api/java/net/URI.html])? If yes and we further force the rule that scheme must be specified with authority,
then the syntax will be
[[scheme:]//authority]path
I think the colons can be distinguished in this case.
I agree that the current patch is simple and clean. However, users cannot perform something like
{code}
> ls
a:b foo
> bin/hadoop fs -put * baz
{code}
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12536917 ]
eric baldeschwieler commented on HADOOP-2066:
---------------------------------------------
I feel like we are going down a rat hole here. How do we get our input syntax to not surprise users? users expect to be able to specify posix like file strings. If URL syntax is going to keep throwing surprises and cross platform issues at us, maybe we need to create a syntax that doesn;t pull in a lot of unwanted baggage?
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2066) filenames with ':' colon throws
java.lang.IllegalArgumentException
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-2066:
-------------------------------------------
Attachment: (was: 2066_20071019.patch)
> filenames with ':' colon throws java.lang.IllegalArgumentException
> ------------------------------------------------------------------
>
> Key: HADOOP-2066
> URL: https://issues.apache.org/jira/browse/HADOOP-2066
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: lohit vijayarenu
> Attachments: 2066_20071022.patch, HADOOP-2066.patch
>
>
> File names containing colon ":" throws java.lang.IllegalArgumentException while LINUX file system supports it.
> $ hadoop dfs -put ./testfile-2007-09-24-03:00:00.gz filenametest
> Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
> URI: testfile-2007-09-24-03:00:00.gz
> at org.apache.hadoop.fs.Path.initialize(Path.java:140)
> at org.apache.hadoop.fs.Path.<init>(Path.java:126)
> at org.apache.hadoop.fs.Path.<init>(Path.java:50)
> at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:273)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:117)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:776)
> at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:757)
> at org.apache.hadoop.fs.FsShell.copyFromLocal(FsShell.java:116)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:1229)
> at org.apache.hadoop.util.ToolBase.doMain(ToolBase.java:187)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:1342)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI: testfile-2007-09-24-03:00:00.gz
> at java.net.URI.checkPath(URI.java:1787)
> at java.net.URI.<init>(URI.java:735)
> at org.apache.hadoop.fs.Path.initialize(Path.java:137)
> ... 10 more
> Path(String pathString) when given a filename which contains ':' treats it as URI and selects anything before ':' as
> scheme, which in this case is clearly not a valid scheme.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.