You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Dmytro Molkov (JIRA)" <ji...@apache.org> on 2010/04/24 01:40:51 UTC
[jira] Created: (HDFS-1109) HFTP and URL Encoding
HFTP and URL Encoding
---------------------
Key: HDFS-1109
URL: https://issues.apache.org/jira/browse/HDFS-1109
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.2, 0.20.1, 0.20.3, 0.21.0, 0.22.0
Reporter: Dmytro Molkov
We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1109:
-----------------------------------------
Component/s: contrib/hdfsproxy
data-node
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860957#action_12860957 ]
sam rash commented on HDFS-1109:
--------------------------------
Ah, I see. I actually agree making the file part of the path is a cleaner approach (put the resource there as the URI spec suggests). I think my question was more about if there was a way to do it w/o changing the HFTP api--I think the url encoding approach would maintain compatibility. A new client could url encode and by default, I believe jetty will url decode the string and hence not require a server change.
that said, i don't object. just throwing the thought out there
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860424#action_12860424 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
This will also be server side only change. There are two servlets that get affected by it the FileDataServlet and StreamFile, since you cannot run cluster with namenode and datanode not synced on build version there is no breaking compatibility.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860778#action_12860778 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
Well, it will essentially serve the same purpose. We could either url encode and decode that parameter being passed to streamFile servlet. Or change it to be url path instead of url parameter. Since all other HFTP operations pass the path of the file as the url path changing the streaming servlet to follow the same logic.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860753#action_12860753 ]
sam rash commented on HDFS-1109:
--------------------------------
just curious, why not url encode the +?
http://blooberry.com/indexdot/html/topics/urlencoding.htm
%2B would result in '+'
(w/o examples, maybe I don't see the precise problem, sorry)
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891197#action_12891197 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1109:
----------------------------------------------
Hi Rohini, the bug you discovered is not y0.20.1xx specific. Trunk also has this bug. So it is better create a new JIRA in order to fix it since this was already resolved in May.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.2_y0.20.1xx_incremental.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-1109:
--------------------------------
Attachment: HDFS-1109.2.patch
Reattaching the patch with license header in the new test.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-1109:
--------------------------------
Status: Patch Available (was: Open)
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.20.1, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-1109:
--------------------------------
Status: Patch Available (was: Open)
Submitting for tests.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.20.1, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1109:
-----------------------------------------
Attachment: HDFS-1109.2_y0.20.1xx.patch
HDFS-1109.2_y0.20.1xx.patch: for y0.20.1xx
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12860990#action_12860990 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
This url request is being generated in the namenode as a redirect for client. Client first contacts the namenode with a path to read, and this path is not url decoded when namenode performs its operations (I was getting exceptions with file not existing when encoding + by hand), then the client is forwarded to a datanode with a new url where the path is passed as a parameter and it is not decoded either, so it is either way namenode + datanode servlets modification.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891504#action_12891504 ]
Rohini Palaniswamy commented on HDFS-1109:
------------------------------------------
Thanks. Created issue https://issues.apache.org/jira/browse/HDFS-1317 for the fix in HDFSProxy.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.2_y0.20.1xx_incremental.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861287#action_12861287 ]
Hadoop QA commented on HDFS-1109:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442924/HDFS-1109.2.patch
against trunk revision 937914.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/328/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/328/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/328/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/328/console
This message is automatically generated.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-1109:
-----------------------------------
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: 0.22.0
Resolution: Fixed
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861239#action_12861239 ]
Hadoop QA commented on HDFS-1109:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12442918/HDFS-1109.patch
against trunk revision 937914.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 release audit. The applied patch generated 113 release audit warnings (more than the trunk's current 112 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/327/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/327/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/327/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/327/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/327/console
This message is automatically generated.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated HDFS-1109:
-------------------------------------
Attachment: HDFS-1109.2_y0.20.1xx_incremental.patch
HDFS-1109.2_y0.20.1xx_incremental.patch: for y0.20.1xx. This is an incremental patch over Nicholas'. It fixes an issue in hdfsproxy where the filepath in streamFile is not picked up.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.2_y0.20.1xx_incremental.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861630#action_12861630 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
I just checked HDFS-330 and HDFS-964 also fail the same hdfsproxy test. I cannot figure out why though. This means that 1109 patch is not affecting the testcase.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861497#action_12861497 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
I cannot see the exact failure of hdfsproxy test, but it succeeds on my local machine.
Can someone please review the patch?
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-1109:
--------------------------------
Attachment: HDFS-1109.patch
Attaching a patch. It also affects HDFSProxy code, would like someone who knows about that part to review this one.
It also introduces TestHftpFileSystem unittest to test the specific problem this patch is solving, but since there was no test for hftp before it should serve as an initial test for Hftp in general.
The patch also fixes the problem introduced by HDFS-991 which threw NPE when trying to read a file through HFTP (stream file servlet was getting name.conf from the datanode context)
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "sam rash (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861644#action_12861644 ]
sam rash commented on HDFS-1109:
--------------------------------
+1 I reviewed the patch and lgtm
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888140#action_12888140 ]
Tsz Wo (Nicholas), SZE commented on HDFS-1109:
----------------------------------------------
This change is similar to changing DatanodeProtocol. We definitely mark it as an incompatible change if DatanodeProtocol is changed.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12888129#action_12888129 ]
Dmytro Molkov commented on HDFS-1109:
-------------------------------------
This change is only incompatible if:
1) Namenode and datanode are of different versions
2) you are going directly to datanode
Otherwise since this is a server only change client and server can be
of different (pre fix/post fix) versions
On Jul 13, 2010, at 5:00 PM, "Tsz Wo (Nicholas), SZE (JIRA)"
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865032#action_12865032 ]
dhruba borthakur commented on HDFS-1109:
----------------------------------------
Code looks good, I wil commit this by the end of this week unless anybody else has any review comments.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HDFS-1109:
-----------------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed])
Marking this as incompatible change since it changed the internal url query.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/hdfsproxy, data-node
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.2_y0.20.1xx.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov reassigned HDFS-1109:
-----------------------------------
Assignee: Dmytro Molkov
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1109) HFTP and URL Encoding
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865874#action_12865874 ]
dhruba borthakur commented on HDFS-1109:
----------------------------------------
I just resolved this. Thanks Dmytro.
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Fix For: 0.22.0
>
> Attachments: HDFS-1109.2.patch, HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1109) HFTP and URL Encoding
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-1109:
--------------------------------
Status: Open (was: Patch Available)
> HFTP and URL Encoding
> ---------------------
>
> Key: HDFS-1109
> URL: https://issues.apache.org/jira/browse/HDFS-1109
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 0.20.2, 0.20.1, 0.20.3, 0.21.0, 0.22.0
> Reporter: Dmytro Molkov
> Assignee: Dmytro Molkov
> Attachments: HDFS-1109.patch
>
>
> We just saw this error happen in our cluster. If there is a file that has a "+" sign in the name it is not readable through HFTP protocol.
> The problem is when we are reading a file with HFTP we are passing a name of the file as a parameter in request and + gets undecoded into space on the server side. So the datanode receiving the streamFile request tries to access a file with space instead of + in the name and doesn't find that file.
> The proposed solution is to pass the filename as a part of URL as with all the other HFTP commands, since this is the only place where it is not being treated this way. Are there any objections to this?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.