You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Chris Nauroth (JIRA)" <ji...@apache.org> on 2016/06/17 21:50:05 UTC
[jira] [Comment Edited] (HADOOP-13287)
TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains
'+'.
[ https://issues.apache.org/jira/browse/HADOOP-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337042#comment-15337042 ]
Chris Nauroth edited comment on HADOOP-13287 at 6/17/16 9:49 PM:
-----------------------------------------------------------------
My test run was on branch-2.8 against an S3 bucket in US-west-2. What I saw happening was a double decoding in {{S3xLoginHelper#extractLoginDetails}}:
{code}
public static Login extractLoginDetails(URI name) {
try {
String authority = name.getAuthority();
...
String password = URLDecoder.decode(login.substring(loginSplit + 1),
"UTF-8");
{code}
According to the JavaDocs for [{{URI#getAuthority}}|http://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getAuthority()], it performs decoding already on the output. Then we do a second explicit decoding by calling {{URLDecoder#decode}}. First, {{getAuthority}} translates "%2B" to "\+". Then, {{URLDecoder#decode}} translates "\+" to " ", which isn't correct for the credentials.
However, this appear to be only a problem in the JUnit test runs. I also built a distro and tested manually with URIs that contain '\+' encoded as "%2B", and that worked just fine. The reason it works fine there is because of different encoding rules applied by round-tripping through a {{Path}} before the {{FileSystem#get}} call gets triggered. With {{Path}}, the '\+' gets double-encoded to "%252B", so double-decoding at the S3A layer is correct logic.
To make this work, the test should follow the same encoding as would be used on the CLI. The attached path switches from constructing a {{URI}} to constructing a {{Path}}. I switched the exception stifling logic to catch {{IllegalArgumentException}}, becauses that's what {{Path}} throws. With this, the test passes with a secret containing a '+'. [~stevel@apache.org] or [~raviprak], I understand one of you might have a secret with a '/' from your work on HADOOP-3733. Would you mind testing this patch to make sure the test still passes with '/'?
was (Author: cnauroth):
My test run was on branch-2.8 against an S3 bucket in US-west-2. What I saw happening was a double decoding in {{S3xLoginHelper#extractLoginDetails}}:
{code}
public static Login extractLoginDetails(URI name) {
try {
String authority = name.getAuthority();
...
String password = URLDecoder.decode(login.substring(loginSplit + 1),
"UTF-8");
{code}
According to the JavaDocs for [{{URI#getAuthority}}|http://docs.oracle.com/javase/7/docs/api/java/net/URI.html#getAuthority()], it performs decoding already on the output. Then we do a second explicit decoding by calling {{URLDecoder#decode}}. First, {{getAuthority}} translates "%2B" to "\+". Then, {{URLDecoder#decode}} translates "\+" to " ", which isn't correct for the credentials.
However, this appear to be only a problem in the JUnit test runs. I also built a distro and tested manually with URIs that contain '+' encoded as "%2B", and that worked just fine. The reason it works fine there is because of different encoding rules applied by round-tripping through a {{Path}} before the {{FileSystem#get}} call gets triggered. With {{Path}}, the '+' gets double-encoded to "%252B", so double-decoding at the S3A layer is correct logic.
To make this work, the test should follow the same encoding as would be used on the CLI. The attached path switches from constructing a {{URI}} to constructing a {{Path}}. I switched the exception stifling logic to catch {{IllegalArgumentException}}, becauses that's what {{Path}} throws. With this, the test passes with a secret containing a '+'. [~stevel@apache.org] or [~raviprak], I understand one of you might have a secret with a '/' from your work on HADOOP-3733. Would you mind testing this patch to make sure the test still passes with '/'?
> TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains '+'.
> -------------------------------------------------------------------------------
>
> Key: HADOOP-13287
> URL: https://issues.apache.org/jira/browse/HADOOP-13287
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Reporter: Chris Nauroth
> Assignee: Chris Nauroth
> Priority: Minor
> Attachments: HADOOP-13287.001.patch
>
>
> HADOOP-3733 fixed accessing S3A with credentials on the command line for an AWS secret key containing a '/'. The patch added a new test suite: {{TestS3ACredentialsInURL}}. One of the tests fails if your AWS secret key contains a '+'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org