You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@jclouds.apache.org by "Erik Paulsson (JIRA)" <ji...@apache.org> on 2013/09/11 23:08:54 UTC
[jira] [Commented] (JCLOUDS-278) SwiftClient parse exception on '%'
symbols
[ https://issues.apache.org/jira/browse/JCLOUDS-278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13764778#comment-13764778 ]
Erik Paulsson commented on JCLOUDS-278:
---------------------------------------
I did some more testing today involving straight REST calls using cURL to bypass any JClouds interaction.
My up front conclusion is that JClouds should not be doing any URL decoding of REST response data since response data is never encoded.
Below are my findings testing against SDSC Swift storage:
I started off with the following 6 files already in my container. These files were previously stored by our OpenStackStorageProvider class using JClouds SwiftClient:
Original container listing:
AND_-_100%_TVX_EXT/test_update.txt
NassauGrouper.jpg
NiceDolphin.jpg
NiceLobster.jpg
SnooksKing.jpg
blankπ × { } © 佈 б ' ' —.txt
PUT/store a new file using unencoded ID:
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" -X PUT -d "testing\n123" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>/AND_-_100%_TVX_EXT/test2_update.txt
HTTP/1.1 201 Created
Last-Modified: Wed, 11 Sep 2013 19:42:58 GMT
Content-Length: 0
Etag: f8444d02c4dc8d8b685c74152d19b9b2
Content-Type: text/html; charset=UTF-8
Date: Wed, 11 Sep 2013 19:42:58 GMT
ubuntu@<aws>:~$
ubuntu@<aws>:~$
List my container to see the new file:
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>
HTTP/1.1 200 OK
X-Container-Meta-Acl-User-Write: root:
Content-Length: 173
X-Container-Object-Count: 7
X-Container-Meta-Space-Total-Size: 0
X-Container-Meta-Space-Created: 2013-08-27T14:42:41
Accept-Ranges: bytes
X-Container-Meta-Space-Count: 0
X-Timestamp: 1377628963.70059
X-Container-Bytes-Used: 1114294
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Sep 2013 19:44:01 GMT
AND_-_100%_TVX_EXT/test2_update.txt
AND_-_100%_TVX_EXT/test_update.txt
NassauGrouper.jpg
NiceDolphin.jpg
NiceLobster.jpg
SnooksKing.jpg
blankπ × { } © 佈 б ' ' —.txt
ubuntu@<aws>:~$
ubuntu@<aws>:~$
ubuntu@<aws>:~$
GET the new file using its unencoded ID:
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>/AND_-_100%_TVX_EXT/test2_update.txt
HTTP/1.1 200 OK
Content-Length: 12
Accept-Ranges: bytes
Last-Modified: Wed, 11 Sep 2013 19:42:58 GMT
Etag: f8444d02c4dc8d8b685c74152d19b9b2
X-Timestamp: 1378928578.72758
Content-Type: application/x-www-form-urlencoded
Date: Wed, 11 Sep 2013 19:44:57 GMT
testing\n123
ubuntu@<aws>:~$
ubuntu@<aws>:~$
ubuntu@<aws>:~$
PUT/store using the same ID, but encoded (Notice I updated the data being stored):
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" -X PUT -d "testing\n123\nupdate using encoded ID" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>/AND_-_100%25_TVX_EXT%2Ftest2_update.txt
HTTP/1.1 201 Created
Last-Modified: Wed, 11 Sep 2013 19:47:03 GMT
Content-Length: 0
Etag: 72607f63e68b22218c6586007c983e91
Content-Type: text/html; charset=UTF-8
Date: Wed, 11 Sep 2013 19:47:03 GMT
ubuntu@<aws>:~$
ubuntu@<aws>:~$
List the files again in my container to make sure a new object wasn't created since I used the encoded ID (Nope, same 7 files):
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>
HTTP/1.1 200 OK
X-Container-Meta-Acl-User-Write: root:
Content-Length: 173
X-Container-Object-Count: 7
X-Container-Meta-Space-Total-Size: 0
X-Container-Meta-Space-Created: 2013-08-27T14:42:41
Accept-Ranges: bytes
X-Container-Meta-Space-Count: 0
X-Timestamp: 1377628963.70059
X-Container-Bytes-Used: 1114319
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Sep 2013 19:47:11 GMT
AND_-_100%_TVX_EXT/test2_update.txt
AND_-_100%_TVX_EXT/test_update.txt
NassauGrouper.jpg
NiceDolphin.jpg
NiceLobster.jpg
SnooksKing.jpg
blankπ × { } © 佈 б ' ' —.txt
ubuntu@<aws>:~$
ubuntu@<aws>:~$
I can GET the newly created file using the encoded ID:
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>/AND_-_100%25_TVX_EXT%2Ftest2_update.txt
HTTP/1.1 200 OK
Content-Length: 37
Accept-Ranges: bytes
Last-Modified: Wed, 11 Sep 2013 19:47:03 GMT
Etag: 72607f63e68b22218c6586007c983e91
X-Timestamp: 1378928823.78157
Content-Type: application/x-www-form-urlencoded
Date: Wed, 11 Sep 2013 19:48:08 GMT
testing\n123\nupdate using encoded ID
ubuntu@<aws>:~$
ubuntu@<aws>:~$
AND I can GET the same newly created file using the unencoded ID:
ubuntu@<aws>:~$ curl -D - -H "X-Auth-Token: <MY_AUTH_TOKEN>" https://<subdomain>.auth.cloud.sdsc.edu:443/v1/<account-id>/<container>/AND_-_100%_TVX_EXT/test2_update.txt
HTTP/1.1 200 OK
Content-Length: 37
Accept-Ranges: bytes
Last-Modified: Wed, 11 Sep 2013 19:47:03 GMT
Etag: 72607f63e68b22218c6586007c983e91
X-Timestamp: 1378928823.78157
Content-Type: application/x-www-form-urlencoded
Date: Wed, 11 Sep 2013 19:49:08 GMT
testing\n123\nupdate using encoded ID
ubuntu@<aws>:~$
> SwiftClient parse exception on '%' symbols
> ------------------------------------------
>
> Key: JCLOUDS-278
> URL: https://issues.apache.org/jira/browse/JCLOUDS-278
> Project: jclouds
> Issue Type: Bug
> Affects Versions: 1.5.10, 1.6.2
> Environment: Tested on:
> org.jclouds:jclouds-allblobstore:1.5.5
> AND
> org.apache.jclouds:jclouds-allblobstore:1.6.2-incubating
> Reporter: Erik Paulsson
>
> I've tested this using org.jclouds:jclouds-allblobstore v1.5.5 and 1.6.2-incubating. I get a parse exception using the SwiftClient when object names contain a '%' symbol.
> For example an object whose name is 'AND_-_100%_TVX/test.txt' will get stored fine, but when I attempt to list the contents of the container that the object is in I get the following exception:
> Sep 10, 2013 10:26:40 AM org.jclouds.logging.jdk.JDKLogger logError
> SEVERE: Error parsing input
> java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - For input string: "_T"
> at java.net.URLDecoder.decode(URLDecoder.java:192)
> at org.jclouds.util.Strings2.urlDecode(Strings2.java:97)
> at org.jclouds.http.Uris$UriBuilder.appendPath(Uris.java:157)
> at org.jclouds.openstack.swift.functions.ParseObjectInfoListFromJsonResponse$3.apply(ParseObjectInfoListFromJsonResponse.java:82)
> at org.jclouds.openstack.swift.functions.ParseObjectInfoListFromJsonResponse$3.apply(ParseObjectInfoListFromJsonResponse.java:80)
> at com.google.common.collect.Iterators$9.transform(Iterators.java:893)
> at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
> at com.google.common.collect.Sets.newTreeSet(Sets.java:345)
> at org.jclouds.openstack.swift.functions.ParseObjectInfoListFromJsonResponse.apply(ParseObjectInfoListFromJsonResponse.java:79)
> at org.jclouds.openstack.swift.functions.ParseObjectInfoListFromJsonResponse.apply(ParseObjectInfoListFromJsonResponse.java:54)
> at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:66)
> at org.jclouds.http.functions.ParseJson.apply(ParseJson.java:46)
> at com.google.common.util.concurrent.Futures$3.apply(Futures.java:376)
> at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:518)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> This may affect more than just the SwiftClient API since it looks like the bug may be in:
> org.jclouds.util.Strings2.urlDecode
> org.jclouds.http.Uris$UriBuilder.appendPath
> Here is an example of a simple method that generates the exception:
> private PageSet<ObjectInfo> listObjects(String containerName,
> String prefix,
> int maxResults,
> String marker) {
> ListContainerOptions containerOptions = ListContainerOptions.Builder.maxResults(maxResults);
> if(marker != null) containerOptions.afterMarker(marker);
> if(prefix != null) containerOptions.withPrefix(prefix);
> return swiftClient.listObjects(containerName, containerOptions);
> }
> Calling this method with the container name of the container containing the object whose name contains a '%' symbol and a maxResults of 1000 and null for both prefix and marker will result in the above exception.
> Let's just say my container name is "percent-container" and I have an object in this container with the name "AND_-_100%_TVX/test.txt"
> So this is what the method call to the above method might look like:
> listObjects("percent-container", null, 1000, null);
> Even if I URL encode the object name before storing it I still get the same exception when retrieving a list of contents for the space.
> I can run a cURL command straight to the Swift storage provider to get the object and it returns fine.
> Am I doing anything wrong? Do I need to handle '%' symbols in a special way? The docs swift docs don't express any character restrictions on object names: http://docs.openstack.org/trunk/openstack-object-storage/admin/content/containers-and-objects.html
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira