You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Jordan (JIRA)" <ji...@apache.org> on 2010/09/01 03:42:53 UTC
[jira] Created: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Url encoded strings never get decoded? How do we fix this?
----------------------------------------------------------
Key: THRIFT-885
URL: https://issues.apache.org/jira/browse/THRIFT-885
Project: Thrift
Issue Type: Bug
Reporter: Jordan
This is for the test java server that was written to test the client.
Suppose you have a method such as testString(theTestString)
If you call it from the javascript like so:
client.testString("hello man");
The string that your server implementation sees will be all crazy and url encoded.
public static void testString(string theTestString) {
// Inside here that string looks like "hello%20man".
}
We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
I'm not exactly sure what will work, but the following seemed to work mostly okay, but there are still bugs. I basically just strip out encoded parts from the entire message (even though the problem seems to only exist inside string fields.)
Around line 115.
<code>
HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
byte[] entityContent = EntityUtils.toByteArray(entity);
String decodedEntityContentString = java.net.URLDecoder.decodeURIComponent(new String(entityContent, "UTF-8"));
byte[] decodedEntityContentBytes = decodedEntityContentString.getBytes("UTF-8");
System.out.println("Incoming content: " + new String(entityContent));
System.out.println("Incoming content normalized: " + new String(decodedEntityContentBytes, "UTF-8"));
// This almost works, except escaped quotes inside of strings
final String output = this.thriftRequest(decodedEntityContentBytes);
System.out.println("Outgoing content: "+output);
EntityTemplate body = new EntityTemplate(new ContentProducer() {
</code>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani updated THRIFT-885:
----------------------------------
Fix Version/s: 0.5
Affects Version/s: 0.4
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905246#action_12905246 ]
Roger Meier commented on THRIFT-885:
------------------------------------
yes, I see... have to integrate that with your testcode.
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910771#action_12910771 ]
Bryan Duxbury commented on THRIFT-885:
--------------------------------------
Is this ready for commit?
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_jordans_patch_and_testcase.patch, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904964#action_12904964 ]
Jordan commented on THRIFT-885:
-------------------------------
I've attached what I think fixes it. (A simple diff)
So basically, I've concluded that when we don't call encodeURIComponent in the javascript, the TJSONProtocol does it's job (for the most part (see notes about forward slashes)) but our client does not. Our javascript clients don't speak proper json (except only by coincidence when we use encodeURIComponent).
So the fix makes sure that when we write out strings it escapes important characters.
I didn't need to do anything on the reverse direction (reading back results) because, like I said, the server's TJSONProtocol speaks JSON properly, so when it returns strings, it properly escapes them.
I didn't include the modified test case in the diff, but here is what I did to the test case (not much):
*Please* note that the tab is actually a tab character, so when I post this it might just change it into a space.
If someone would like to, you can also add this quickly into the test.html.
var stringTransferTestString = 'quote: " backslash: \\ forwardslash-escaped: \/ ' +
' backspace: \b formfeed: \f newline: \n return: \r tab: ' +
' now-all-of-them-together: "\\\/\b\n\r\t' +
' now-a-bunch-of-junk: !@#$%^&*()(*&^%$#{}{}<><><';
document.write("client.testString() => "+(client.testString(stringTransferTestString) == stringTransferTestString) + "<br>");
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Reporter: Jordan
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> I'm not exactly sure what will work, but the following seemed to work mostly okay, but there are still bugs. I basically just strip out encoded parts from the entire message (even though the problem seems to only exist inside string fields.)
> Around line 115.
> <code>
> HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
> byte[] entityContent = EntityUtils.toByteArray(entity);
> String decodedEntityContentString = java.net.URLDecoder.decodeURIComponent(new String(entityContent, "UTF-8"));
> byte[] decodedEntityContentBytes = decodedEntityContentString.getBytes("UTF-8");
> System.out.println("Incoming content: " + new String(entityContent));
> System.out.println("Incoming content normalized: " + new String(decodedEntityContentBytes, "UTF-8"));
>
> // This almost works, except escaped quotes inside of strings
> final String output = this.thriftRequest(decodedEntityContentBytes);
> System.out.println("Outgoing content: "+output);
>
> EntityTemplate body = new EntityTemplate(new ContentProducer() {
> </code>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jordan updated THRIFT-885:
--------------------------
Attachment: charEscapingFix.diff
I believe this to be the correct solution (except for forward slashes which I think are TJSONProtocol's problem.)
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
> Attachments: charEscapingFix.diff
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904905#action_12904905 ]
Jordan commented on THRIFT-885:
-------------------------------
I just tried removing it and everything works GREAT so far (that I have tested) except for quotes inside of a string.
In fact, the only reason why this was working at all was because you were using encodeUriComponent in javascript which escaped the quotes into %22.
The json spec says that Strings should be escaped with a backslash. So in the current codebase the json objects that are sent across the wire are really only loose adaptations of the objects they are intended to represent.
To fix this we shouldn't use encodeURIComponent, and instead try to use properJSON (because that is actually what TJSONProtocol expects anyways)
The two steps involve:
1. simply removing en/decodeUriComponent from thrift.js.
2. Escaping all of the fields that the json spec requires (as opposed to relying on encodeUriComponent to do it.)
3. Unescaping all of the escaped fields when deserializing.
Number one is dirt simple, numbers two and three I don't really know where to begin.
See the json documentation on Strings here http://json.org/
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Reporter: Jordan
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> I'm not exactly sure what will work, but the following seemed to work mostly okay, but there are still bugs. I basically just strip out encoded parts from the entire message (even though the problem seems to only exist inside string fields.)
> Around line 115.
> <code>
> HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
> byte[] entityContent = EntityUtils.toByteArray(entity);
> String decodedEntityContentString = java.net.URLDecoder.decodeURIComponent(new String(entityContent, "UTF-8"));
> byte[] decodedEntityContentBytes = decodedEntityContentString.getBytes("UTF-8");
> System.out.println("Incoming content: " + new String(entityContent));
> System.out.println("Incoming content normalized: " + new String(decodedEntityContentBytes, "UTF-8"));
>
> // This almost works, except escaped quotes inside of strings
> final String output = this.thriftRequest(decodedEntityContentBytes);
> System.out.println("Outgoing content: "+output);
>
> EntityTemplate body = new EntityTemplate(new ContentProducer() {
> </code>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904872#action_12904872 ]
Jordan commented on THRIFT-885:
-------------------------------
Do we even need to call encodeURIComponent in thrift.js?
>From what I understand, encodeURIComponent is primarily for ensuring one thing - That key value pairs encoded in urls don't get messed up by user supplied text.
Since this is sent over post, in json format, it shouldn't matter right?
Can't we just skip the uriEncoding step?
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Reporter: Jordan
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> I'm not exactly sure what will work, but the following seemed to work mostly okay, but there are still bugs. I basically just strip out encoded parts from the entire message (even though the problem seems to only exist inside string fields.)
> Around line 115.
> <code>
> HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
> byte[] entityContent = EntityUtils.toByteArray(entity);
> String decodedEntityContentString = java.net.URLDecoder.decodeURIComponent(new String(entityContent, "UTF-8"));
> byte[] decodedEntityContentBytes = decodedEntityContentString.getBytes("UTF-8");
> System.out.println("Incoming content: " + new String(entityContent));
> System.out.println("Incoming content normalized: " + new String(decodedEntityContentBytes, "UTF-8"));
>
> // This almost works, except escaped quotes inside of strings
> final String output = this.thriftRequest(decodedEntityContentBytes);
> System.out.println("Outgoing content: "+output);
>
> EntityTemplate body = new EntityTemplate(new ContentProducer() {
> </code>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905302#action_12905302 ]
T Jake Luciani commented on THRIFT-885:
---------------------------------------
Hi Jordan, this looks good. I think this is the only reasonable way todo it.
Let me run some tests and I'll commit.
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roger Meier updated THRIFT-885:
-------------------------------
Attachment: THRIFT-885_testcase_only.patch
thanks for committing this!
could you please add the testcase provided above as well?
I've just added the Testcase as a seperate file now
[^THRIFT-885_testcase_only.patch]
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_jordans_patch_and_testcase.patch, THRIFT-885_remove_encodeURIComponent.patch, THRIFT-885_testcase_only.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roger Meier updated THRIFT-885:
-------------------------------
Attachment: THRIFT-885_remove_encodeURIComponent.patch
Hi Jordan!
Yes, this is a bug.
on wire Thrift is just UTF-8 not URL encoded
I did some tests with FireFox on Debian Lenny and IE7 on Windows XP
{code}
THttpServer.cpp: Server side output is
testString("Afrikaans%2C%20Alemannisch .....
instead of UTF-8
testString("Afrikaans, Alemannisch .....
{code}
The attached patch fixes that issue.
By the way...update your lib/js/ to the latest version! It contains a enhanced Unit Test based on QUnit, that's a good helper for finding bugs!
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roger Meier updated THRIFT-885:
-------------------------------
Attachment: THRIFT-885_jordans_patch_and_testcase.patch
It works fine for me, attached is Jordan's patch and the corresponding unittest within test.html.
see http://www.bufferoverflow.ch:8088/test/test.html
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_jordans_patch_and_testcase.patch, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-885) Url encoded strings never get decoded?
How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jordan updated THRIFT-885:
--------------------------
Description:
This is for the test java server that was written to test the client.
Suppose you have a method such as testString(theTestString)
If you call it from the javascript like so:
client.testString("hello man");
The string that your server implementation sees will be all crazy and url encoded.
public static void testString(string theTestString) {
// Inside here that string looks like "hello%20man".
}
We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
was:
This is for the test java server that was written to test the client.
Suppose you have a method such as testString(theTestString)
If you call it from the javascript like so:
client.testString("hello man");
The string that your server implementation sees will be all crazy and url encoded.
public static void testString(string theTestString) {
// Inside here that string looks like "hello%20man".
}
We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
I'm not exactly sure what will work, but the following seemed to work mostly okay, but there are still bugs. I basically just strip out encoded parts from the entire message (even though the problem seems to only exist inside string fields.)
Around line 115.
<code>
HttpEntity entity = ((HttpEntityEnclosingRequest) request).getEntity();
byte[] entityContent = EntityUtils.toByteArray(entity);
String decodedEntityContentString = java.net.URLDecoder.decodeURIComponent(new String(entityContent, "UTF-8"));
byte[] decodedEntityContentBytes = decodedEntityContentString.getBytes("UTF-8");
System.out.println("Incoming content: " + new String(entityContent));
System.out.println("Incoming content normalized: " + new String(decodedEntityContentBytes, "UTF-8"));
// This almost works, except escaped quotes inside of strings
final String output = this.thriftRequest(decodedEntityContentBytes);
System.out.println("Outgoing content: "+output);
EntityTemplate body = new EntityTemplate(new ContentProducer() {
</code>
Component/s: JavaScript - Compiler
deleting proposed fix, see later post.
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani reassigned THRIFT-885:
-------------------------------------
Assignee: T Jake Luciani
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "Jordan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12905215#action_12905215 ]
Jordan commented on THRIFT-885:
-------------------------------
Roger,
I noted in one of my comments that I tried your fix and it didn't work. Please take a look at my patch. I already tried what you have posted as a patch and it doesn't work when you have nested quotes inside the text (or newlines etc.)
My patch deals with these by escaping them into standard json strings in accordance with www.json.org.
I even made a test case to ensure that it works properly. Can you please take a look at it and integrate it into the source code?
Thanks.
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Reporter: Jordan
> Attachments: charEscapingFix.diff, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (THRIFT-885) Url encoded strings never get
decoded? How do we fix this?
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
T Jake Luciani resolved THRIFT-885.
-----------------------------------
Resolution: Fixed
Comitted with some minor edits.
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_jordans_patch_and_testcase.patch, THRIFT-885_remove_encodeURIComponent.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (THRIFT-885) Url encoded strings never
get decoded? How do we fix this?
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12910904#action_12910904 ]
Roger Meier edited comment on THRIFT-885 at 9/22/10 1:16 AM:
-------------------------------------------------------------
thanks for committing this!
could you please add the testcase provided above as well?
I've just added the Testcase as a seperate file now
[^THRIFT-885_testcase_only.patch]
=> moved test case to THRIFT-913
was (Author: roger.meier):
thanks for committing this!
could you please add the testcase provided above as well?
I've just added the Testcase as a seperate file now
[^THRIFT-885_testcase_only.patch]
> Url encoded strings never get decoded? How do we fix this?
> ----------------------------------------------------------
>
> Key: THRIFT-885
> URL: https://issues.apache.org/jira/browse/THRIFT-885
> Project: Thrift
> Issue Type: Bug
> Components: JavaScript - Compiler
> Affects Versions: 0.4
> Reporter: Jordan
> Assignee: T Jake Luciani
> Fix For: 0.5
>
> Attachments: charEscapingFix.diff, THRIFT-885_jordans_patch_and_testcase.patch, THRIFT-885_remove_encodeURIComponent.patch, THRIFT-885_testcase_only.patch
>
>
> This is for the test java server that was written to test the client.
> Suppose you have a method such as testString(theTestString)
> If you call it from the javascript like so:
> client.testString("hello man");
> The string that your server implementation sees will be all crazy and url encoded.
> public static void testString(string theTestString) {
> // Inside here that string looks like "hello%20man".
> }
> We need it to be urlencoded when coming across the wire because we're using JSON as the protocol, but once it gets to thrift server code, it needs to be normal.
> So what's going on here? Inside of strings (of javascript objects) things like quotes and spaces are escaped. This is fine, but I do not believe that is part of the TJSONProtocol. This is something that is specific to having a js thrift client am I right? So where do we fix this?
> Edit: I had proposed a code fix which I do not believe to be the correct solution. Deleting- see future post for what I believe to be correct.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.