You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Kai Griffin - Relaxing <ka...@griffinbyteworks.com> on 2009/05/17 10:17:59 UTC
URL encoding issue
Hi,
These past few weeks, I've been re-building our flagship product as a
Flex application, with couchdb as the back end, and this has been
progressing really well. However, I've hit a strange snag, to do with
URL encoding. Flex's HTTPService component automatically url-encodes
requests, so that...
....?startkey=["SYD","LAX","2008-01-01T00:00:00Z"]&endkey=["SYD","LAX",2008-01-01T23:59:59Z"]
becomes...
....?startkey=[%22SYD%22,%22LAX%22,%222008-01-01T00:00:00Z%22]&endkey=[%22SYD%22,%22LAX%22,....etc....]
that is, when I look at the Erlang console after the request has been
sent from Flex, I see the request with quotes encoded as %22's as shown
above when issued from Flex.
The trouble is, couchdb fails to return any rows when those quotes are
encoded as %22's like that. When I issue the exact same request in
curl, the quotes are not automatically encoded (they still show as
quotes in Erlang's console), and I get the correct result set for the
view. I always use group=true, so that's not the difference.
What is particularly strange to me, is that a different design/view I
have works perfectly with the following key:
....?startkey=[911,%222008-01-01T00:00:00Z%22]&...etc
In this case, the first item in the key is an unquoted integer, but
couchdb is still fine with the quoted date, and correctly returns a
result from either Flex or curl.
Any suggestions from a kind soul out there?
Cheers,
Kai
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
Ah - very good to know! Thanks, Chris.
-Kai
On 20/05/2009 00:21, Chris Anderson wrote:
> On Tue, May 19, 2009 at 12:07 PM, Kai Griffin <ka...@griffinbyteworks.com> wrote:
>
>> Mind you, I would no doubt
>> feel a lot more strongly about the x-http-method-override idea if I needed
>> DELETE; in my application, remote clients will not be deleting stuff - at
>> best, records will be flagged as "deleted", but left in place.
>>
>
> bulk_docs can handle DELETE just fine, as it's just a matter of adding
> {"_deleted" : true} to your document
>
> Chris
>
>
Re: URL encoding issue
Posted by Chris Anderson <jc...@apache.org>.
On Tue, May 19, 2009 at 12:07 PM, Kai Griffin <ka...@griffinbyteworks.com> wrote:
> Mind you, I would no doubt
> feel a lot more strongly about the x-http-method-override idea if I needed
> DELETE; in my application, remote clients will not be deleting stuff - at
> best, records will be flagged as "deleted", but left in place.
bulk_docs can handle DELETE just fine, as it's just a matter of adding
{"_deleted" : true} to your document
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
That's an interesting discussion. I can't say I'm as concerned about
using POST to _bulk_docs in lieu of PUT as some, but that's probably a
reflection of my relatively recent move into the RESTful world, not to
mention to couchdb itself. I guess I'm just struggling to understand
why one is preferable to the other, even if the other is represents a
work-around. In any case, _bulk_docs certainly works for me :-) Mind
you, I would no doubt feel a lot more strongly about the
x-http-method-override idea if I needed DELETE; in my application,
remote clients will not be deleting stuff - at best, records will be
flagged as "deleted", but left in place. The only work around I can
think for the lack of DELETE in my http client would be to write an
external process that gets launched via couchdb's external http handlers
mechanism to handle the deleting for me - that's certainly clunky, but
would get the job done.
While discussing problematic clients like Flash, couchdb's lack of
support for the need of a crossdomain policy file for certain clients
such as Flash and Silverlight had me tearing my hair out for awhile (I
have been evaluating both, though have since tossed out Silverlight).
The biggest problem is Silverlight, because it expects the crossdomain
file to not only exist on the remote server, but also to exist through
the same port (ie, 5984). In the end, I got it working by duplicating
the erlang code that handles favicon.ico requests to make a new handler
for requests to /crossdomain.xml . Rather fun, as it was my first
exposure to erlang... but obviously very much a hack!
Cheers,
Kai
On 19/05/2009 17:12, Jared Scheel wrote:
> We've been discussing this issue here as well:
> http://www.mail-archive.com/dev@couchdb.apache.org/msg02730.html
>
> I think that couchdb could really benefit from supporting an override
> instead of relying on a workaround with _bulk_docs though. The
> x-http-method-override header has been used in several APIs, including
> all of Google's, to get around this limitation in a clean fashion.
>
> -Jared
>
>
> On Tue, May 19, 2009 at 3:04 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
>> On 19 May 2009, at 02:11, Kai Griffin wrote:
>>
>>
>>> Hi Sam, the Flex HTTPService only supports two content-types -
>>> x-www-form-urlencoded and xml - but the former of these works fine. My
>>> application won't need DELETE, fortunately. As for PUT, I'd forgotten about
>>> that limitation - I suppose I could POST to _bulk_docs as a work-around?
>>>
>> That works.
>>
>> Cheers
>> Jan
>> --
>>
>>
>>
>>> Fortunately, the client-side of the application (that is, at the Flex
>>> end) consists mainly of read-only operations, and most of the writing to
>>> database is handled by processes located on the server-side over night. I'm
>>> also using the external process feature of couch to launch legacy code on
>>> the server which has been cannibalised from our old app, which performs
>>> lengthy bulk record update operations.
>>>
>>> Many thanks for pointing these out.
>>>
>>> By the way, the mystery view issue that originally prompted this thread is
>>> now resolved; thanks, all, for the help.
>>>
>>> Kai
>>>
>>> On 19/05/2009 01:32, Samuel Wan wrote:
>>>
>>>> Do you need to set the json content-type headers in the HTTPService
>>>> requests?
>>>>
>>>> BTW, the flash player doesn't support delete or put, will that be a
>>>> problem for you?
>>>>
>>>> -Sam
>>>>
>>>> On Mon, May 18, 2009 at 1:35 PM, Kai Griffin <ka...@griffinbyteworks.com>
>>>> wrote:
>>>>
>>>>
>>>>> On 18/05/2009 21:54, Rhett Sutphin wrote:
>>>>>
>>>>>
>>>>>> Hi Kai,
>>>>>>
>>>>>> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Hi Brian,
>>>>>>>
>>>>>>> I've been escaping the square brackets and double quotes in curl, like
>>>>>>> this \[\" ....\" \] - which seems to work. I'll start using -g in
>>>>>>> curl so
>>>>>>> I don't have to escape the square braces.
>>>>>>>
>>>>>>> Wrapping the url in single quotes results in the single quote being
>>>>>>> interpreted as part of the url itself, and curl complains that
>>>>>>> "protocol
>>>>>>> 'http is not supported".
>>>>>>>
>>>>>>> As for %22's... I cannot make that work at all. If I encode double
>>>>>>> quotes as %22's in curl, couchdb complains about invalid JSON, and it
>>>>>>> looks
>>>>>>> as if all the %'s have been stripped out and the 22's become part of
>>>>>>> the
>>>>>>> request. For example the following request:
>>>>>>>
>>>>>>> (snip)
>>>>>>>
>>>>>>>
>>>>>> You are on windows, right? In the regular windows shell, environment
>>>>>> variable substitution is done with %ENV_VAR_NAME% (roughly equivalent
>>>>>> to
>>>>>> $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of the
>>>>>> 22SYD environment variable and concatenate 22". If there is no such
>>>>>> env
>>>>>> var, the result would just be "22". This fits with the invalid JSON
>>>>>> couchdb
>>>>>> is seeing.
>>>>>>
>>>>>> Long story short, I think you need to escape the '%'s from the windows
>>>>>> shell. I don't know how to do this, but I'm sure it is possible.
>>>>>>
>>>>>>
>>>>> Aha! Thank-you, Rhett - this never crossed my mind, though it makes
>>>>> perfect
>>>>> sense. It's a simple matter of doubling the % (%%) to ensure the
>>>>> command
>>>>> processor doesn't try to do the replacement. When I just tried this, my
>>>>> curl command worked fine, and as Brian suggested, I get a successful
>>>>> result
>>>>> whether I escape the quotes with backslashes or encoded them as %22's.
>>>>> Now
>>>>> I just need to figure out why the HTTPService from Flex is not
>>>>> working...
>>>>> clearly the url-encoding has nothing to do with it and I've no doubt got
>>>>> an
>>>>> unrelated bug in my own work.
>>>>>
>>>>> Best regards,
>>>>> Kai
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Rhett
>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>
>
Re: URL encoding issue
Posted by Jared Scheel <ja...@gmail.com>.
We've been discussing this issue here as well:
http://www.mail-archive.com/dev@couchdb.apache.org/msg02730.html
I think that couchdb could really benefit from supporting an override
instead of relying on a workaround with _bulk_docs though. The
x-http-method-override header has been used in several APIs, including
all of Google's, to get around this limitation in a clean fashion.
-Jared
On Tue, May 19, 2009 at 3:04 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On 19 May 2009, at 02:11, Kai Griffin wrote:
>
>> Hi Sam, the Flex HTTPService only supports two content-types -
>> x-www-form-urlencoded and xml - but the former of these works fine. My
>> application won't need DELETE, fortunately. As for PUT, I'd forgotten about
>> that limitation - I suppose I could POST to _bulk_docs as a work-around?
>
> That works.
>
> Cheers
> Jan
> --
>
>
>> Fortunately, the client-side of the application (that is, at the Flex
>> end) consists mainly of read-only operations, and most of the writing to
>> database is handled by processes located on the server-side over night. I'm
>> also using the external process feature of couch to launch legacy code on
>> the server which has been cannibalised from our old app, which performs
>> lengthy bulk record update operations.
>>
>> Many thanks for pointing these out.
>>
>> By the way, the mystery view issue that originally prompted this thread is
>> now resolved; thanks, all, for the help.
>>
>> Kai
>>
>> On 19/05/2009 01:32, Samuel Wan wrote:
>>>
>>> Do you need to set the json content-type headers in the HTTPService
>>> requests?
>>>
>>> BTW, the flash player doesn't support delete or put, will that be a
>>> problem for you?
>>>
>>> -Sam
>>>
>>> On Mon, May 18, 2009 at 1:35 PM, Kai Griffin <ka...@griffinbyteworks.com>
>>> wrote:
>>>
>>>> On 18/05/2009 21:54, Rhett Sutphin wrote:
>>>>
>>>>> Hi Kai,
>>>>>
>>>>> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>>>>>
>>>>>
>>>>>> Hi Brian,
>>>>>>
>>>>>> I've been escaping the square brackets and double quotes in curl, like
>>>>>> this \[\" ....\" \] - which seems to work. I'll start using -g in
>>>>>> curl so
>>>>>> I don't have to escape the square braces.
>>>>>>
>>>>>> Wrapping the url in single quotes results in the single quote being
>>>>>> interpreted as part of the url itself, and curl complains that
>>>>>> "protocol
>>>>>> 'http is not supported".
>>>>>>
>>>>>> As for %22's... I cannot make that work at all. If I encode double
>>>>>> quotes as %22's in curl, couchdb complains about invalid JSON, and it
>>>>>> looks
>>>>>> as if all the %'s have been stripped out and the 22's become part of
>>>>>> the
>>>>>> request. For example the following request:
>>>>>>
>>>>>> (snip)
>>>>>>
>>>>> You are on windows, right? In the regular windows shell, environment
>>>>> variable substitution is done with %ENV_VAR_NAME% (roughly equivalent
>>>>> to
>>>>> $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of the
>>>>> 22SYD environment variable and concatenate 22". If there is no such
>>>>> env
>>>>> var, the result would just be "22". This fits with the invalid JSON
>>>>> couchdb
>>>>> is seeing.
>>>>>
>>>>> Long story short, I think you need to escape the '%'s from the windows
>>>>> shell. I don't know how to do this, but I'm sure it is possible.
>>>>>
>>>> Aha! Thank-you, Rhett - this never crossed my mind, though it makes
>>>> perfect
>>>> sense. It's a simple matter of doubling the % (%%) to ensure the
>>>> command
>>>> processor doesn't try to do the replacement. When I just tried this, my
>>>> curl command worked fine, and as Brian suggested, I get a successful
>>>> result
>>>> whether I escape the quotes with backslashes or encoded them as %22's.
>>>> Now
>>>> I just need to figure out why the HTTPService from Flex is not
>>>> working...
>>>> clearly the url-encoding has nothing to do with it and I've no doubt got
>>>> an
>>>> unrelated bug in my own work.
>>>>
>>>> Best regards,
>>>> Kai
>>>>
>>>>
>>>>
>>>>> Rhett
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
Re: URL encoding issue
Posted by Jan Lehnardt <ja...@apache.org>.
On 19 May 2009, at 02:11, Kai Griffin wrote:
> Hi Sam, the Flex HTTPService only supports two content-types - x-www-
> form-urlencoded and xml - but the former of these works fine. My
> application won't need DELETE, fortunately. As for PUT, I'd
> forgotten about that limitation - I suppose I could POST to
> _bulk_docs as a work-around?
That works.
Cheers
Jan
--
> Fortunately, the client-side of the application (that is, at the
> Flex end) consists mainly of read-only operations, and most of the
> writing to database is handled by processes located on the server-
> side over night. I'm also using the external process feature of
> couch to launch legacy code on the server which has been
> cannibalised from our old app, which performs lengthy bulk record
> update operations.
>
> Many thanks for pointing these out.
>
> By the way, the mystery view issue that originally prompted this
> thread is now resolved; thanks, all, for the help.
>
> Kai
>
> On 19/05/2009 01:32, Samuel Wan wrote:
>> Do you need to set the json content-type headers in the HTTPService
>> requests?
>>
>> BTW, the flash player doesn't support delete or put, will that be a
>> problem for you?
>>
>> -Sam
>>
>> On Mon, May 18, 2009 at 1:35 PM, Kai Griffin <kai@griffinbyteworks.com
>> > wrote:
>>
>>> On 18/05/2009 21:54, Rhett Sutphin wrote:
>>>
>>>> Hi Kai,
>>>>
>>>> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>>>>
>>>>
>>>>> Hi Brian,
>>>>>
>>>>> I've been escaping the square brackets and double quotes in
>>>>> curl, like
>>>>> this \[\" ....\" \] - which seems to work. I'll start using -
>>>>> g in curl so
>>>>> I don't have to escape the square braces.
>>>>>
>>>>> Wrapping the url in single quotes results in the single quote
>>>>> being
>>>>> interpreted as part of the url itself, and curl complains that
>>>>> "protocol
>>>>> 'http is not supported".
>>>>>
>>>>> As for %22's... I cannot make that work at all. If I encode
>>>>> double
>>>>> quotes as %22's in curl, couchdb complains about invalid JSON,
>>>>> and it looks
>>>>> as if all the %'s have been stripped out and the 22's become
>>>>> part of the
>>>>> request. For example the following request:
>>>>>
>>>>> (snip)
>>>>>
>>>> You are on windows, right? In the regular windows shell,
>>>> environment
>>>> variable substitution is done with %ENV_VAR_NAME% (roughly
>>>> equivalent to
>>>> $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value
>>>> of the
>>>> 22SYD environment variable and concatenate 22". If there is no
>>>> such env
>>>> var, the result would just be "22". This fits with the invalid
>>>> JSON couchdb
>>>> is seeing.
>>>>
>>>> Long story short, I think you need to escape the '%'s from the
>>>> windows
>>>> shell. I don't know how to do this, but I'm sure it is possible.
>>>>
>>> Aha! Thank-you, Rhett - this never crossed my mind, though it
>>> makes perfect
>>> sense. It's a simple matter of doubling the % (%%) to ensure the
>>> command
>>> processor doesn't try to do the replacement. When I just tried
>>> this, my
>>> curl command worked fine, and as Brian suggested, I get a
>>> successful result
>>> whether I escape the quotes with backslashes or encoded them as
>>> %22's. Now
>>> I just need to figure out why the HTTPService from Flex is not
>>> working...
>>> clearly the url-encoding has nothing to do with it and I've no
>>> doubt got an
>>> unrelated bug in my own work.
>>>
>>> Best regards,
>>> Kai
>>>
>>>
>>>
>>>> Rhett
>>>>
>>>>
>>>
>>
>>
>
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
Hi Sam, the Flex HTTPService only supports two content-types -
x-www-form-urlencoded and xml - but the former of these works fine. My
application won't need DELETE, fortunately. As for PUT, I'd forgotten
about that limitation - I suppose I could POST to _bulk_docs as a
work-around? Fortunately, the client-side of the application (that is,
at the Flex end) consists mainly of read-only operations, and most of
the writing to database is handled by processes located on the
server-side over night. I'm also using the external process feature of
couch to launch legacy code on the server which has been cannibalised
from our old app, which performs lengthy bulk record update operations.
Many thanks for pointing these out.
By the way, the mystery view issue that originally prompted this thread
is now resolved; thanks, all, for the help.
Kai
On 19/05/2009 01:32, Samuel Wan wrote:
> Do you need to set the json content-type headers in the HTTPService requests?
>
> BTW, the flash player doesn't support delete or put, will that be a
> problem for you?
>
> -Sam
>
> On Mon, May 18, 2009 at 1:35 PM, Kai Griffin <ka...@griffinbyteworks.com> wrote:
>
>> On 18/05/2009 21:54, Rhett Sutphin wrote:
>>
>>> Hi Kai,
>>>
>>> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>>>
>>>
>>>> Hi Brian,
>>>>
>>>> I've been escaping the square brackets and double quotes in curl, like
>>>> this \[\" ....\" \] - which seems to work. I'll start using -g in curl so
>>>> I don't have to escape the square braces.
>>>>
>>>> Wrapping the url in single quotes results in the single quote being
>>>> interpreted as part of the url itself, and curl complains that "protocol
>>>> 'http is not supported".
>>>>
>>>> As for %22's... I cannot make that work at all. If I encode double
>>>> quotes as %22's in curl, couchdb complains about invalid JSON, and it looks
>>>> as if all the %'s have been stripped out and the 22's become part of the
>>>> request. For example the following request:
>>>>
>>>> (snip)
>>>>
>>> You are on windows, right? In the regular windows shell, environment
>>> variable substitution is done with %ENV_VAR_NAME% (roughly equivalent to
>>> $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of the
>>> 22SYD environment variable and concatenate 22". If there is no such env
>>> var, the result would just be "22". This fits with the invalid JSON couchdb
>>> is seeing.
>>>
>>> Long story short, I think you need to escape the '%'s from the windows
>>> shell. I don't know how to do this, but I'm sure it is possible.
>>>
>> Aha! Thank-you, Rhett - this never crossed my mind, though it makes perfect
>> sense. It's a simple matter of doubling the % (%%) to ensure the command
>> processor doesn't try to do the replacement. When I just tried this, my
>> curl command worked fine, and as Brian suggested, I get a successful result
>> whether I escape the quotes with backslashes or encoded them as %22's. Now
>> I just need to figure out why the HTTPService from Flex is not working...
>> clearly the url-encoding has nothing to do with it and I've no doubt got an
>> unrelated bug in my own work.
>>
>> Best regards,
>> Kai
>>
>>
>>
>>> Rhett
>>>
>>>
>>
>
>
Re: URL encoding issue
Posted by Samuel Wan <sa...@samuelwan.com>.
Do you need to set the json content-type headers in the HTTPService requests?
BTW, the flash player doesn't support delete or put, will that be a
problem for you?
-Sam
On Mon, May 18, 2009 at 1:35 PM, Kai Griffin <ka...@griffinbyteworks.com> wrote:
> On 18/05/2009 21:54, Rhett Sutphin wrote:
>>
>> Hi Kai,
>>
>> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>>
>>> Hi Brian,
>>>
>>> I've been escaping the square brackets and double quotes in curl, like
>>> this \[\" ....\" \] - which seems to work. I'll start using -g in curl so
>>> I don't have to escape the square braces.
>>>
>>> Wrapping the url in single quotes results in the single quote being
>>> interpreted as part of the url itself, and curl complains that "protocol
>>> 'http is not supported".
>>>
>>> As for %22's... I cannot make that work at all. If I encode double
>>> quotes as %22's in curl, couchdb complains about invalid JSON, and it looks
>>> as if all the %'s have been stripped out and the 22's become part of the
>>> request. For example the following request:
>>>
>>> (snip)
>>
>> You are on windows, right? In the regular windows shell, environment
>> variable substitution is done with %ENV_VAR_NAME% (roughly equivalent to
>> $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of the
>> 22SYD environment variable and concatenate 22". If there is no such env
>> var, the result would just be "22". This fits with the invalid JSON couchdb
>> is seeing.
>>
>> Long story short, I think you need to escape the '%'s from the windows
>> shell. I don't know how to do this, but I'm sure it is possible.
>
> Aha! Thank-you, Rhett - this never crossed my mind, though it makes perfect
> sense. It's a simple matter of doubling the % (%%) to ensure the command
> processor doesn't try to do the replacement. When I just tried this, my
> curl command worked fine, and as Brian suggested, I get a successful result
> whether I escape the quotes with backslashes or encoded them as %22's. Now
> I just need to figure out why the HTTPService from Flex is not working...
> clearly the url-encoding has nothing to do with it and I've no doubt got an
> unrelated bug in my own work.
>
> Best regards,
> Kai
>
>
>>
>> Rhett
>>
>
>
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
On 18/05/2009 21:54, Rhett Sutphin wrote:
> Hi Kai,
>
> On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
>
>> Hi Brian,
>>
>> I've been escaping the square brackets and double quotes in curl,
>> like this \[\" ....\" \] - which seems to work. I'll start using
>> -g in curl so I don't have to escape the square braces.
>>
>> Wrapping the url in single quotes results in the single quote being
>> interpreted as part of the url itself, and curl complains that
>> "protocol 'http is not supported".
>>
>> As for %22's... I cannot make that work at all. If I encode double
>> quotes as %22's in curl, couchdb complains about invalid JSON, and it
>> looks as if all the %'s have been stripped out and the 22's become
>> part of the request. For example the following request:
>>
>> (snip)
>
> You are on windows, right? In the regular windows shell, environment
> variable substitution is done with %ENV_VAR_NAME% (roughly equivalent
> to $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of
> the 22SYD environment variable and concatenate 22". If there is no
> such env var, the result would just be "22". This fits with the
> invalid JSON couchdb is seeing.
>
> Long story short, I think you need to escape the '%'s from the windows
> shell. I don't know how to do this, but I'm sure it is possible.
Aha! Thank-you, Rhett - this never crossed my mind, though it makes
perfect sense. It's a simple matter of doubling the % (%%) to ensure
the command processor doesn't try to do the replacement. When I just
tried this, my curl command worked fine, and as Brian suggested, I get a
successful result whether I escape the quotes with backslashes or
encoded them as %22's. Now I just need to figure out why the
HTTPService from Flex is not working... clearly the url-encoding has
nothing to do with it and I've no doubt got an unrelated bug in my own work.
Best regards,
Kai
>
> Rhett
>
Re: URL encoding issue
Posted by Rhett Sutphin <rh...@detailedbalance.net>.
Hi Kai,
On May 18, 2009, at 2:40 PM, Kai Griffin wrote:
> Hi Brian,
> Thanks for your help.
>
> I've been escaping the square brackets and double quotes in curl,
> like this \[\" ....\" \] - which seems to work. I'll start using -
> g in curl so I don't have to escape the square braces.
>
> Wrapping the url in single quotes results in the single quote being
> interpreted as part of the url itself, and curl complains that
> "protocol 'http is not supported".
>
> As for %22's... I cannot make that work at all. If I encode double
> quotes as %22's in curl, couchdb complains about invalid JSON, and
> it looks as if all the %'s have been stripped out and the 22's
> become part of the request. For example the following request:
>
> curl -g -X GET http: (...url...)?startkey=[%22SYD%22,%22LAX
> %22,%222008-01-01T00:00:00Z%22%]&endkey=[%22SYD%22,%22LAX
> %22,%222008-01-05T23:59:59Z
> %22]&group=true&stale=ok
>
> ....results in the following error from couchdb:
>
> {"error":"invalid_json","reason":"[22,22,-01-01T00:00:00Z]"}
You are on windows, right? In the regular windows shell, environment
variable substitution is done with %ENV_VAR_NAME% (roughly equivalent
to $ENV_VAR_NAME in bash). So %22SYD%22 would mean "take the value of
the 22SYD environment variable and concatenate 22". If there is no
such env var, the result would just be "22". This fits with the
invalid JSON couchdb is seeing.
Long story short, I think you need to escape the '%'s from the windows
shell. I don't know how to do this, but I'm sure it is possible.
Rhett
>
>
> Replacing all the %22's with escaped double quotes (like this:
> \" ) works perfectly; I get the rows I was expecting.
>
> Somewhere along the line, those %'s are being stripped out and the
> request mangled. I will fire up my linux laptop later and see if
> any of these problems exist there.
>
> Cheers,
> Kai
>
> On 18/05/2009 13:46, Brian Candler wrote:
>> On Sun, May 17, 2009 at 10:17:59AM +0200, Kai Griffin - Relaxing
>> wrote:
>>
>>> The trouble is, couchdb fails to return any rows when those quotes
>>> are encoded as %22's like that.
>>>
>>
>> Are you sure? Can you tun the same curl command line twice, once
>> where it
>> works, and once which is identical except you replace every double
>> quote
>> with %22 ?
>>
>> Note 1: make sure the whole URL goes inside single quotes:
>>
>> curl 'http://127.0.0.1:5984/.....'
>>
>> Note 2: be careful with square brackets and curl, as it will try to
>> perform
>> its own globbing on them. Either use the -g flag to curl, or encode
>> your
>> square brackets as %5B and %5D.
>>
>> HTH,
>>
>> Brian.
>>
>>
>
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
Hi Brian,
Thanks for your help.
I've been escaping the square brackets and double quotes in curl, like
this \[\" ....\" \] - which seems to work. I'll start using -g in
curl so I don't have to escape the square braces.
Wrapping the url in single quotes results in the single quote being
interpreted as part of the url itself, and curl complains that "protocol
'http is not supported".
As for %22's... I cannot make that work at all. If I encode double
quotes as %22's in curl, couchdb complains about invalid JSON, and it
looks as if all the %'s have been stripped out and the 22's become part
of the request. For example the following request:
curl -g -X GET http:
(...url...)?startkey=[%22SYD%22,%22LAX%22,%222008-01-01T00:00:00Z%22%]&endkey=[%22SYD%22,%22LAX%22,%222008-01-05T23:59:59Z
%22]&group=true&stale=ok
....results in the following error from couchdb:
{"error":"invalid_json","reason":"[22,22,-01-01T00:00:00Z]"}
Replacing all the %22's with escaped double quotes (like this: \" )
works perfectly; I get the rows I was expecting.
Somewhere along the line, those %'s are being stripped out and the
request mangled. I will fire up my linux laptop later and see if any of
these problems exist there.
Cheers,
Kai
On 18/05/2009 13:46, Brian Candler wrote:
> On Sun, May 17, 2009 at 10:17:59AM +0200, Kai Griffin - Relaxing wrote:
>
>> The trouble is, couchdb fails to return any rows when those quotes are
>> encoded as %22's like that.
>>
>
> Are you sure? Can you tun the same curl command line twice, once where it
> works, and once which is identical except you replace every double quote
> with %22 ?
>
> Note 1: make sure the whole URL goes inside single quotes:
>
> curl 'http://127.0.0.1:5984/.....'
>
> Note 2: be careful with square brackets and curl, as it will try to perform
> its own globbing on them. Either use the -g flag to curl, or encode your
> square brackets as %5B and %5D.
>
> HTH,
>
> Brian.
>
>
Re: URL encoding issue
Posted by Brian Candler <B....@pobox.com>.
On Sun, May 17, 2009 at 10:17:59AM +0200, Kai Griffin - Relaxing wrote:
> The trouble is, couchdb fails to return any rows when those quotes are
> encoded as %22's like that.
Are you sure? Can you tun the same curl command line twice, once where it
works, and once which is identical except you replace every double quote
with %22 ?
Note 1: make sure the whole URL goes inside single quotes:
curl 'http://127.0.0.1:5984/.....'
Note 2: be careful with square brackets and curl, as it will try to perform
its own globbing on them. Either use the -g flag to curl, or encode your
square brackets as %5B and %5D.
HTH,
Brian.
Re: URL encoding issue
Posted by Kai Griffin <ka...@griffinbyteworks.com>.
Thank-you, Adam. I'll keep playing with it. I wonder if running
couchdb (v0.9) in Windows has anything to do with it. I'm using Windows
because I also want to make use of the couchdb External handler to run
chunks of legacy code on certain result sets.
I actually typed out those examples by hand, so that leading
double-quote is not missing in reality (but well-spotted). In any case,
I'm not getting an error - just a result with zero rows.
Cheers,
Kai
On 17/05/2009 15:52, Adam Kocoloski wrote:
> Hi Kai, I don't think the URL-encoded double quotes are your problem.
> I just tried encoding all the double-quotes with curl and it worked
> fine -- even when using a compound key and a string as the first
> element of that key.
>
> If you can write up a reproducible test case that shows unexpected
> behavior I'm sure someone here can take a crack at it. Best,
>
> Adam
>
> P.S. you missed a leading double-quote in your first query string.
> Could it be that simple?
>
> On May 17, 2009, at 4:17 AM, Kai Griffin - Relaxing wrote:
>
>> Hi,
>> These past few weeks, I've been re-building our flagship product as a
>> Flex application, with couchdb as the back end, and this has been
>> progressing really well. However, I've hit a strange snag, to do
>> with URL encoding. Flex's HTTPService component automatically
>> url-encodes requests, so that...
>>
>> ....?startkey=["SYD","LAX","2008-01-01T00:00:00Z"]&endkey=["SYD","LAX",2008-01-01T23:59:59Z"]
>>
>>
>> becomes...
>>
>> ....?startkey=[%22SYD%22,%22LAX%22,%222008-01-01T00:00:00Z%22]&endkey=[%22SYD%22,%22LAX%22,....etc....]
>>
>>
>> that is, when I look at the Erlang console after the request has been
>> sent from Flex, I see the request with quotes encoded as %22's as
>> shown above when issued from Flex.
>>
>> The trouble is, couchdb fails to return any rows when those quotes
>> are encoded as %22's like that. When I issue the exact same request
>> in curl, the quotes are not automatically encoded (they still show as
>> quotes in Erlang's console), and I get the correct result set for the
>> view. I always use group=true, so that's not the difference.
>>
>> What is particularly strange to me, is that a different design/view I
>> have works perfectly with the following key:
>>
>> ....?startkey=[911,%222008-01-01T00:00:00Z%22]&...etc
>>
>> In this case, the first item in the key is an unquoted integer, but
>> couchdb is still fine with the quoted date, and correctly returns a
>> result from either Flex or curl.
>>
>> Any suggestions from a kind soul out there?
>>
>> Cheers,
>> Kai
>
>
Re: URL encoding issue
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Kai, I don't think the URL-encoded double quotes are your problem.
I just tried encoding all the double-quotes with curl and it worked
fine -- even when using a compound key and a string as the first
element of that key.
If you can write up a reproducible test case that shows unexpected
behavior I'm sure someone here can take a crack at it. Best,
Adam
P.S. you missed a leading double-quote in your first query string.
Could it be that simple?
On May 17, 2009, at 4:17 AM, Kai Griffin - Relaxing wrote:
> Hi,
> These past few weeks, I've been re-building our flagship product as
> a Flex application, with couchdb as the back end, and this has been
> progressing really well. However, I've hit a strange snag, to do
> with URL encoding. Flex's HTTPService component automatically url-
> encodes requests, so that...
>
> ....?
> startkey=["SYD","LAX","2008-01-01T00:00:00Z"]&endkey=["SYD","LAX",
> 2008-01-01T23:59:59Z"]
>
> becomes...
>
> ....?startkey=[%22SYD%22,%22LAX%22,%222008-01-01T00:00:00Z
> %22]&endkey=[%22SYD%22,%22LAX%22,....etc....]
>
> that is, when I look at the Erlang console after the request has
> been sent from Flex, I see the request with quotes encoded as %22's
> as shown above when issued from Flex.
>
> The trouble is, couchdb fails to return any rows when those quotes
> are encoded as %22's like that. When I issue the exact same request
> in curl, the quotes are not automatically encoded (they still show
> as quotes in Erlang's console), and I get the correct result set for
> the view. I always use group=true, so that's not the difference.
>
> What is particularly strange to me, is that a different design/view
> I have works perfectly with the following key:
>
> ....?startkey=[911,%222008-01-01T00:00:00Z%22]&...etc
>
> In this case, the first item in the key is an unquoted integer, but
> couchdb is still fine with the quoted date, and correctly returns a
> result from either Flex or curl.
>
> Any suggestions from a kind soul out there?
>
> Cheers,
> Kai