You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by Jesse Glick <no...@github.com> on 2018/07/10 03:51:57 UTC

[jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

[JCLOUDS-1401](https://jira.apache.org/jira/browse/JCLOUDS-1401)

I managed to write integration tests for this in Jenkins (which fail without this patch and pass with it). I would have liked to add a test case for it to `AWSS3ClientLiveTest` (or `S3ClientLiveTest`? not sure I grasp the distinction) but did not manage to get the existing tests to run; I either got errors about an unknown access key, or a 403, regardless of what parameters I tried; and there is no apparent way to pick up values from `~/.aws/credentials`. [Testing](https://cwiki.apache.org/confluence/display/JCLOUDS/Testing) makes no specific mention of S3 and what the `identity` and `credential` would be, concretely; [Getting Started](https://jclouds.apache.org/guides/aws/) does not give format examples or give a hint about how the internal properties map would look; and developer guides in the source tree like [that for Google Cloud](https://github.com/jclouds/jclouds/blob/1bd3b8f9dfa148e3d3f9a3b13ce3335e620276e1/providers/google-cloud-storage/README.md#running-the-live-tests) seem to be absent for S3.

@carlossg @kuisathaverat
You can view, comment on, or merge this pull request online at:

  https://github.com/jclouds/jclouds/pull/1226

-- Commit Summary --

  * [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters.

-- File Changes --

    M apis/s3/src/main/java/org/jclouds/s3/filters/Aws4SignerBase.java (2)

-- Patch Links --

https://github.com/jclouds/jclouds/pull/1226.patch
https://github.com/jclouds/jclouds/pull/1226.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
jglick commented on this pull request.

@nacx please take another look. I added both a unit test and a live test. A mock test seems redundant given those.

> @@ -507,6 +538,23 @@ public void testCopyWithMetadata() throws InterruptedException, ExecutionExcepti
       }
    }
 
+   // JCLOUDS-1401
+   public void testUnusualKeyCharacters() throws InterruptedException, ExecutionException, TimeoutException, IOException {
+      String containerName = getContainerName();
+      try {
+         String dirName = "a%2Fb&xxx#?:$'\\\"<>čॐ";
+         String fileName = "foo%3Abar.xml";
+         addToContainerAndValidate(containerName, dirName + '/' + fileName);
+         PageSet<? extends StorageMetadata> list = view.getBlobStore().list(containerName,
+                  ListContainerOptions.Builder.prefix(dirName + "/"));
+         assertEquals(list.size(), 1);
+         StorageMetadata md = list.iterator().next();

This fails with the reported AWS error if you s/rawQuery/query/.

> @@ -83,6 +84,11 @@
          + "SignedHeaders=content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class, "
          + "Signature=090f1bb1db984221ae1a20c5d12a82820a0d74b4be85f20daa1431604f41df08";
 
+   private static final String LIST_BUCKET_RESULT = "AWS4-HMAC-SHA256 "
+         + "Credential=AKIAPAEBI3QI4EXAMPLE/20150203/cn-north-1/s3/aws4_request, "
+         + "SignedHeaders=host;x-amz-content-sha256;x-amz-date, "
+         + "Signature=6cc5d0758e2599be7cb172fd57cefab2828201a2b4d372972a83dc304de93958";

Merely based on running the test input against rawQuery. I did not attempt to consult the AWS documentation and manually construct the signature base as per [these instructions](https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html).

The key to the fix is that `Queries.parseKeyValueFromStringToDecodedMap` is used to URL-decode the parameters of the query string after splitting on `&`, and we then reëncode in `getCanonicalizedQueryString`, so using `URI.getQuery()` (which decodes the value) not only double-decodes, it actually loses information. For example,

```
URI u = new URI("https://x.com/?p1=a&p2=b%26c%3Dd&p3=e");
System.out.println(u.getRawQuery());
System.out.println(u.getQuery());
```

will print

```
p1=a&p2=b%26c%3Dd&p3=e
p1=a&p2=b&c=d&p3=e
```

>     public S3ClientLiveTest() {
       this.provider = "s3";
    }
    
+   @Override
+   protected Properties setupProperties() {
+      Properties overrides = super.setupProperties();
+      sessionToken = setIfTestSystemPropertyPresent(overrides, provider + ".sessionToken");
+      return overrides;
+   }
+
+   @Override
+   protected ContextBuilder newBuilder() {
+      ContextBuilder builder = super.newBuilder();
+      if (sessionToken != null) {
+         builder.credentialsSupplier(new Supplier<Credentials>() {
+            @Override
+            public Credentials get() {
+               return SessionCredentials.builder().identity(identity).credential(credential).sessionToken(sessionToken).build();

Without this I was unable to run live tests at all. (Maybe something about IAM?) The new parameter is optional.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#pullrequestreview-136273302

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Ignasi Barrera <no...@github.com>.
Thanks, @jglick!

I'd say before running live tests you could add a small unit test that just verifies the `createStringToSign` method so we don't break it accidentally in the future.

You can also add a mock unit test to the [S3ClientMockTest](https://github.com/jclouds/jclouds/blob/master/apis/s3/src/test/java/org/jclouds/s3/S3ClientMockTest.java) class. Mock tests just mock HTTP requests and provide a convenient way to do assertions on the generated requests. You could write one test that verifies the signature that has been sent for a request.

For the live tests, we don't mention explicitly the S3 parameters in the wiki because they are the same for all providers. You can invoke the live tests as follows (note this follows the generic format explained in the testing page):

```bash
mvn clean install -Plive -pl :s3 -Dtest=S3ClientLiveTest \
   -Dtest.s3.identity=<access key> \
   -Dtest.s3.credential=<access secret> \
   -Dtest.s3.endpoint=https://s3.amazonaws.com
```

Since S3 is a [generic API](https://jclouds.apache.org/start/concepts/) (if you read this quick page you'll get the difference between the S3 *api* and the AWS-S3 *provider*) that can be implemented by many providers, we need to set the endpoint parameter indicating where to run the live tests against.

Since you are already using AWS, you could run the `aws-s3` provider live tests too, as follows:

```bash
mvn clean install -Plive -pl :aws-s3 -Dtest=AWSS3ClientLiveTest \
   -Dtest.aws-s3.identity=<acces key> \
   -Dtest.aws-s3.credential=<access secret>
```

Note that since you are now using the `aws-s3` provider there is no need to provide the endpoint, as the AWS provider does already know where it needs to connect.

The credentials you need to pass in the identity and credential property are the same ones you pass to the jclouds `ContextBuilder` when creating the context, and in general, the additional properties you have to pass to the tests in the command line, are those extra properties you configure when creating the context.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-403902292

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Ignasi Barrera <no...@github.com>.
Closed #1226.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#event-1728411664

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
I did try running `AWSS3ClientLiveTest` with the arguments you mention, and it did not work, as noted in the PR description. Of course the failure messages were pretty opaque for security reasons. Noting that these parameters are identical to that used by `ContextBuilder` is not particularly helpful to me since our tests do not require such keys to be passed in explicitly—the code picks up `~/.aws/credentials` automatically and calls `credentialsSupplier` on a `SessionCredentials` ultimately using the AWS SDK, and that works fine, and there is no apparent system property corresponding to the `sessionToken`. So I will guess I will have to find a colleague with better understanding of AWS authentication to assist me.

It is good to at least have confirmation that `identity` is an “access key” while `credential` is an “access secret” in AWS terminology—this is what would be useful to clarify in a `README.md`.

I am less confident in the value of a unit test or mock test since it is not self-evident what the actual designed signature would be. The fix is just that which produces a signature that Amazon in fact accepts.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-403940474

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Carlos Sanchez <no...@github.com>.
Thanks Ignasi!

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-404406356

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Carlos Sanchez <no...@github.com>.
carlossg approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#pullrequestreview-136283174

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Oleg Nenashev <no...@github.com>.
oleg-nenashev approved this pull request.

Looks good



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#pullrequestreview-136290748

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
I think I figured out that the existing tests just do not support session credentials (and I cannot access my account without them), so I need to patch the tests.

> since you are now using the aws-s3 provider there is no need to provide the endpoint, as the AWS provider does already know where it needs to connect.

This is not true—the inherited `testCopyCannedAccessPolicyPublic` and `testPutCannedAccessPolicyPublic` fail with an NPE when it is omitted.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-403972004

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Ignasi Barrera <no...@github.com>.
nacx approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#pullrequestreview-136323635

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
@jglick pushed 2 commits.

3ad84f1  Live test demonstrating effectiveness of JCLOUDS-1401 fix.
793b728  Pinning down JCLOUDS-1401 fix behavior with a unit test which is able to distinguish URI.query from URI.rawQuery.


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1226/files/069e1097bb9cc9247a2a0b736ba17a50aa08d0d2..793b728d49c27402ca1b403555e2e02e3cdee72b

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Ignasi Barrera <no...@github.com>.
Thanks for the patch and for taking your time to run the live tests @jglick! Merged to master and 2.1.x.

I'll have a look at the failing tests, as if you run just the S3 ones and pass the endpoint, all succeed. There must be some minor mess about the endpoints in the aws-s3 tests.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-404234793

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
@nacx

> if you run just the S3 ones and pass the endpoint, all succeed

Confirmed that when running `S3ClientLiveTest` directly in the `s3` module, with the endpoint defined, all tests pass including `testCopyCannedAccessPolicyPublic` and `testPutCannedAccessPolicyPublic`. So there must be something amiss with how the endpoint property is defaulted in the `aws-s3` module’s POM ([here](https://github.com/jclouds/jclouds/blob/4958f1f29a8e692319ed89d774009c602fcfc44a/providers/aws-s3/pom.xml#L38) and [here](https://github.com/jclouds/jclouds/blob/4958f1f29a8e692319ed89d774009c602fcfc44a/providers/aws-s3/pom.xml#L136)). I can only confirm that the property is absent from `System.getProperties()` unless you pass ` -Dtest.aws-s3.endpoint=https://s3.amazonaws.com` on the command line.

The other two failures I mentioned were in `AWSS3ClientLiveTest` and I suspect are caused by my running using session credentials defined in the US-East region. Just a guess—my grasp of AWS is limited and certainly does not include the nuances of region selection.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-404262521

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Ivan Fernandez Calvo <no...@github.com>.
kuisathaverat approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#pullrequestreview-135736156

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
There are some tests that fail even without my patch:

```
org.jclouds.aws.AWSResponseException: request PUT https://s3.amazonaws.com/jglick-blobstore-TestBucket HTTP/1.1 failed with code 400, error: AWSError{requestId='…', requestToken='…', code='InvalidBucketName', message='The specified bucket is not valid.', context='{BucketName=jglick-blobstore-TestBucket, HostId=…}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:138)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:107)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy48.putBucketInRegion(Unknown Source)
	at org.jclouds.s3.blobstore.S3BlobStore.createContainerInLocation(S3BlobStore.java:456)
	at org.jclouds.aws.s3.blobstore.AWSS3BlobStore.createContainerInLocation(AWSS3BlobStore.java:105)
	at org.jclouds.s3.blobstore.S3BlobStore.createContainerInLocation(S3BlobStore.java:146)
	at org.jclouds.aws.s3.AWSS3ClientLiveTest.testUseBucketWithUpperCaseName(AWSS3ClientLiveTest.java:109)

org.jclouds.http.HttpResponseException: Server rejected operation connecting to PUT https://jglick-blobstore11eu.s3-eu-central-1.amazonaws.com/test-blob?X-Amz-Security-Token=…&X-Amz-Algorithm=…&X-Amz-Credential=…/eu-central-1/s3/aws4_request&X-Amz-Date=…&X-Amz-Expires=…&X-Amz-SignedHeaders=host&X-Amz-Signature=… HTTP/1.1
	at sun.net.www.protocol.http.HttpURLConnection.expect100Continue(HttpURLConnection.java:1269)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1348)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.writePayloadToConnection(JavaUrlHttpCommandExecutorService.java:295)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:171)
	at org.jclouds.http.internal.JavaUrlHttpCommandExecutorService.convert(JavaUrlHttpCommandExecutorService.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:97)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:91)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:74)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:45)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.handle(DelegatesToInvocationFunction.java:156)
	at org.jclouds.rest.internal.DelegatesToInvocationFunction.invoke(DelegatesToInvocationFunction.java:123)
	at com.sun.proxy.$Proxy49.invoke(Unknown Source)
	at org.jclouds.aws.s3.AWSS3ClientLiveTest.testV4SignatureOps(AWSS3ClientLiveTest.java:192)
```

These seem to be related to the region, perhaps? And:

```
java.lang.NullPointerException: null
	at java.net.URI$Parser.parse(URI.java:3042)
	at java.net.URI.<init>(URI.java:588)
	at java.net.URI.create(URI.java:850)
	at org.jclouds.s3.S3ClientLiveTest.getObjectURL(S3ClientLiveTest.java:139)
	at org.jclouds.s3.S3ClientLiveTest.testCopyCannedAccessPolicyPublic(S3ClientLiveTest.java:176)

java.lang.NullPointerException: null
	at java.net.URI$Parser.parse(URI.java:3042)
	at java.net.URI.<init>(URI.java:588)
	at java.net.URI.create(URI.java:850)
	at org.jclouds.s3.S3ClientLiveTest.getObjectURL(S3ClientLiveTest.java:139)
	at org.jclouds.s3.S3ClientLiveTest.testPutCannedAccessPolicyPublic(S3ClientLiveTest.java:155)
```

which seem to be caused by an undefined `endpoint`, though I am not sure why since `providers/aws-s3/pom.xml` does seem to default it. Anyway, I will ignore these for now.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/jclouds/jclouds/pull/1226#issuecomment-403980911

Re: [jclouds/jclouds] [JCLOUDS-1401] Properly URL-encode the CanonicalQueryString when it contains funny characters (#1226)

Posted by Jesse Glick <no...@github.com>.
@jglick pushed 1 commit.

069e109  Allow S3ClientLiveTest (and AWSS3ClientLiveTest) to use a session token.


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/jclouds/jclouds/pull/1226/files/8022473e25654dbb4d6ab12587efd90027657840..069e1097bb9cc9247a2a0b736ba17a50aa08d0d2