You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@couchdb.apache.org by Lennart Melzer <l....@tu-bs.de> on 2009/11/09 17:54:05 UTC
POST/PUTing binary Attachments to external Handler
Right now, posting or putting binary data to an external handler
fails, since the data is parsed as a UTF-8 String and not treated as
binary. The external handler never gets to do its work, so this might
be an issue with the handling of data sent to an external handler. Am
I misusing the external-handler interface?
Error snippet:
[error] [<0.95.0>] {error_report,<0.30.0>,
{<0.95.0>,std_error,
{mochiweb_socket_server,235,
{child_error,{ucs,{bad_utf8_character_code}}}}}}
Greetings,
Lennart
Re: POST/PUTing binary Attachments to external Handler
Posted by Lennart Melzer <l....@tu-bs.de>.
Mmmh, so the real problem just seems to be, that the external handling
of POST/PUT calls does not handle binary data seperately from String
data. So one has to base64encode those attachments. I'm not really
good enough at erlang coding (yet) to move the code from the usual
couchdb handling (for multipart/form-data) to external handlers, but
that would be really good.
For now I stick with base64 encoding of binary data, which works ok,
since the produced json object sent to my external ruby script does
not contain any offending "chars".
Greetings,
Lennart
On Nov 11, 2009, at 8:29 AM, Lennart Melzer wrote:
> Hi Paul,
>
> I double checked the response of my external ruby script.
>
> I reduced the code to the bare minimum (nearly representing what the
> python example from the wiki looks like)
>
> #!/usr/bin/env ruby
> require 'rubygems'
> require 'json'
>
> def request
> begin
> while line = STDIN.readline
> yield JSON.parse(line)
> end
> rescue EOFError
> return
> end
> end
>
> def respond code = 200, data = {}, headers = {}
> output = "#{{:code => code,:body => data, :headers =>
> headers}.to_json}\n"
> STDOUT.write output
> STDOUT.flush
> end
> request do |line|
> respond 200, "random"
> end
>
> When I do a GET request to the external handler I get "random" back,
> when I do a POST with a simple form I get "random" back, but when
> POSTing an image (by curl) as multipart/form-data it breaks with the
> aforementioned error returning nothing. So since the return value
> seems to work with other types of request, it can't be my external
> handler.
>
> Sorry for bothering, but I really want to get this working. Been
> reading through the code for some time now, but I don't understand
> why couchdb is even calling send_json in the first place.
>
> Greetings,
>
> Lennart
> On Nov 11, 2009, at 12:11 AM, Paul Davis wrote:
>
>> On Tue, Nov 10, 2009 at 5:56 PM, Lennart Melzer <l....@tu-bs.de>
>> wrote:
>>> Hi Paul,
>>>
>>> I'm not trying to return arbitrary binary data,I really just try
>>> to post an
>>> attached file to an external handler. Right now the handler
>>> returns a static
>>> response which works when posting form data (without binary data
>>> in it) and
>>> with get request. It doesn't work when posting a binary file. I
>>> don't think
>>> it's a problem with the handler itself.
>>>
>>> I don't know why the couchdb is trying to encode the binary post
>>> request as
>>> json.
>>>
>>> Greetings,
>>>
>>> Lennart
>>>
>>> On Nov 10, 2009, at 11:26 PM, Paul Davis wrote:
>>>
>>>> On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l.melzer@tu-
>>>> bs.de> wrote:
>>>>>
>>>>> I just looked through the error logs and the code a bit more. I
>>>>> think
>>>>> this
>>>>> passage describes best what seems to happen (though I don't know
>>>>> why)
>>>>>
>>>>> So the stracktrace shows that the mochiweb passes the request to
>>>>> couch_httpd:handle_request where it seems to try to call send_json
>>>>> (whysoever). It all fails when it tries to encode the binary
>>>>> data as a
>>>>> json
>>>>> string. This all seems to happen before couch_httpd_external.erl
>>>>> stuff
>>>>> gets
>>>>> called, which might be able to handle binary data correctly.
>>>>>
>>>>> Here's the log snippet:
>>>>> crasher:
>>>>> initial call: mochiweb_socket_server:acceptor_loop/1
>>>>> pid: <0.1059.0>
>>>>> registered_name: []
>>>>> exception exit: {ucs,{bad_utf8_character_code}}
>>>>> in function xmerl_ucs:from_utf8/1
>>>>> in call from mochijson2:json_encode_string/2
>>>>> in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
>>>>> in call from lists:foldl/3
>>>>> in call from mochijson2:json_encode_proplist/2
>>>>> in call from couch_httpd:send_json/4
>>>>> in call from couch_httpd:handle_request/5
>>>>> in call from mochiweb_http:headers/5
>>>>>
>>>>> Anyone out that knows more about the internals of couchdb than I
>>>>> do? I
>>>>> would
>>>>> be grateful for any advice (gotta learn some erlang programming
>>>>> soonish).
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Lennart
>>>>>
>>>>>
>>>>> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>>>>>
>>>>>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l.melzer@tu-bs.de
>>>>>> >
>>>>>> wrote:
>>>>>>>
>>>>>>> I am running Revision 832477 of the couchdb trunk
>>>>>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>>>>>
>>>>>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l.melzer@tu-bs.de
>>>>>>>> >
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Right now, posting or putting binary data to an external
>>>>>>>>> handler
>>>>>>>>> fails,
>>>>>>>>> since the data is parsed as a UTF-8 String and not treated
>>>>>>>>> as binary.
>>>>>>>>> The
>>>>>>>>> external handler never gets to do its work, so this might be
>>>>>>>>> an issue
>>>>>>>>> with
>>>>>>>>> the handling of data sent to an external handler. Am I
>>>>>>>>> misusing the
>>>>>>>>> external-handler interface?
>>>>>>>>>
>>>>>>>>> Error snippet:
>>>>>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>>>>>> {<0.95.0>,std_error,
>>>>>>>>> {mochiweb_socket_server,235,
>>>>>>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>>>>>
>>>>>>>>> Greetings,
>>>>>>>>>
>>>>>>>>> Lennart
>>>>>>>>>
>>>>>>>>
>>>>>>>> Lennart,
>>>>>>>>
>>>>>>>> What version of CouchDB are you on? I seem to remember a
>>>>>>>> patch for
>>>>>>>> this going into trunk at some point. Not sure if that was pre/
>>>>>>>> post
>>>>>>>> 0.10 though.
>>>>>>>>
>>>>>>>> Paul Davis
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Lennart,
>>>>>>
>>>>>> Hmm, looking through the svn log I'm not seeing a commit that
>>>>>> mentions
>>>>>> body or posts. I swear I remember something like that coming
>>>>>> through
>>>>>> at one point. Benoit, didn't you have something to do with that?
>>>>>>
>>>>>> Paul Davis
>>>>>
>>>>>
>>>>
>>>> Lennart,
>>>>
>>>> That stack trace suggests that you're trying to return a JSON
>>>> response
>>>> that is not valid JSON. If your external is trying to return
>>>> arbitrary
>>>> binary data in the _external response object you'll need to make
>>>> sure
>>>> and use the body member.
>>>>
>>>> Paul Davis
>>>
>>>
>>
>> Lennart,
>>
>> What are you returning from your _external handler? You should be
>> returning something like:
>>
>> {
>> "code": 200,
>> "body": "arbitrary string of data and some stuff",
>> "headers": {"Content-Type": "application/my-stuff"}
>> }
>>
>> I'd have to remind myself on the exact rules, but returning {"code":
>> 200, "json": "stuff"} will send "stuff" through the JSON serializer,
>> which means it has to conform to unicode escaping rules that
>> mochijson2 exposes via xmerl:ucs16_to_char or whatever.
>>
>> Perhaps you can log the response you're sending and paste a copy of
>> that?
>>
>> HTH,
>> Paul Davis
>
Re: POST/PUTing binary Attachments to external Handler
Posted by Lennart Melzer <l....@tu-bs.de>.
Hi Paul,
I double checked the response of my external ruby script.
I reduced the code to the bare minimum (nearly representing what the
python example from the wiki looks like)
#!/usr/bin/env ruby
require 'rubygems'
require 'json'
def request
begin
while line = STDIN.readline
yield JSON.parse(line)
end
rescue EOFError
return
end
end
def respond code = 200, data = {}, headers = {}
output = "#{{:code => code,:body => data, :headers =>
headers}.to_json}\n"
STDOUT.write output
STDOUT.flush
end
request do |line|
respond 200, "random"
end
When I do a GET request to the external handler I get "random" back,
when I do a POST with a simple form I get "random" back, but when
POSTing an image (by curl) as multipart/form-data it breaks with the
aforementioned error returning nothing. So since the return value
seems to work with other types of request, it can't be my external
handler.
Sorry for bothering, but I really want to get this working. Been
reading through the code for some time now, but I don't understand why
couchdb is even calling send_json in the first place.
Greetings,
Lennart
On Nov 11, 2009, at 12:11 AM, Paul Davis wrote:
> On Tue, Nov 10, 2009 at 5:56 PM, Lennart Melzer <l....@tu-bs.de>
> wrote:
>> Hi Paul,
>>
>> I'm not trying to return arbitrary binary data,I really just try to
>> post an
>> attached file to an external handler. Right now the handler returns
>> a static
>> response which works when posting form data (without binary data in
>> it) and
>> with get request. It doesn't work when posting a binary file. I
>> don't think
>> it's a problem with the handler itself.
>>
>> I don't know why the couchdb is trying to encode the binary post
>> request as
>> json.
>>
>> Greetings,
>>
>> Lennart
>>
>> On Nov 10, 2009, at 11:26 PM, Paul Davis wrote:
>>
>>> On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l.melzer@tu-
>>> bs.de> wrote:
>>>>
>>>> I just looked through the error logs and the code a bit more. I
>>>> think
>>>> this
>>>> passage describes best what seems to happen (though I don't know
>>>> why)
>>>>
>>>> So the stracktrace shows that the mochiweb passes the request to
>>>> couch_httpd:handle_request where it seems to try to call send_json
>>>> (whysoever). It all fails when it tries to encode the binary data
>>>> as a
>>>> json
>>>> string. This all seems to happen before couch_httpd_external.erl
>>>> stuff
>>>> gets
>>>> called, which might be able to handle binary data correctly.
>>>>
>>>> Here's the log snippet:
>>>> crasher:
>>>> initial call: mochiweb_socket_server:acceptor_loop/1
>>>> pid: <0.1059.0>
>>>> registered_name: []
>>>> exception exit: {ucs,{bad_utf8_character_code}}
>>>> in function xmerl_ucs:from_utf8/1
>>>> in call from mochijson2:json_encode_string/2
>>>> in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
>>>> in call from lists:foldl/3
>>>> in call from mochijson2:json_encode_proplist/2
>>>> in call from couch_httpd:send_json/4
>>>> in call from couch_httpd:handle_request/5
>>>> in call from mochiweb_http:headers/5
>>>>
>>>> Anyone out that knows more about the internals of couchdb than I
>>>> do? I
>>>> would
>>>> be grateful for any advice (gotta learn some erlang programming
>>>> soonish).
>>>>
>>>> Greetings,
>>>>
>>>> Lennart
>>>>
>>>>
>>>> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>>>>
>>>>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l.melzer@tu-
>>>>> bs.de>
>>>>> wrote:
>>>>>>
>>>>>> I am running Revision 832477 of the couchdb trunk
>>>>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>>>>
>>>>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l.melzer@tu-bs.de
>>>>>>> >
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Right now, posting or putting binary data to an external
>>>>>>>> handler
>>>>>>>> fails,
>>>>>>>> since the data is parsed as a UTF-8 String and not treated as
>>>>>>>> binary.
>>>>>>>> The
>>>>>>>> external handler never gets to do its work, so this might be
>>>>>>>> an issue
>>>>>>>> with
>>>>>>>> the handling of data sent to an external handler. Am I
>>>>>>>> misusing the
>>>>>>>> external-handler interface?
>>>>>>>>
>>>>>>>> Error snippet:
>>>>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>>>>> {<0.95.0>,std_error,
>>>>>>>> {mochiweb_socket_server,235,
>>>>>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>>>>
>>>>>>>> Greetings,
>>>>>>>>
>>>>>>>> Lennart
>>>>>>>>
>>>>>>>
>>>>>>> Lennart,
>>>>>>>
>>>>>>> What version of CouchDB are you on? I seem to remember a patch
>>>>>>> for
>>>>>>> this going into trunk at some point. Not sure if that was pre/
>>>>>>> post
>>>>>>> 0.10 though.
>>>>>>>
>>>>>>> Paul Davis
>>>>>>
>>>>>>
>>>>>
>>>>> Lennart,
>>>>>
>>>>> Hmm, looking through the svn log I'm not seeing a commit that
>>>>> mentions
>>>>> body or posts. I swear I remember something like that coming
>>>>> through
>>>>> at one point. Benoit, didn't you have something to do with that?
>>>>>
>>>>> Paul Davis
>>>>
>>>>
>>>
>>> Lennart,
>>>
>>> That stack trace suggests that you're trying to return a JSON
>>> response
>>> that is not valid JSON. If your external is trying to return
>>> arbitrary
>>> binary data in the _external response object you'll need to make
>>> sure
>>> and use the body member.
>>>
>>> Paul Davis
>>
>>
>
> Lennart,
>
> What are you returning from your _external handler? You should be
> returning something like:
>
> {
> "code": 200,
> "body": "arbitrary string of data and some stuff",
> "headers": {"Content-Type": "application/my-stuff"}
> }
>
> I'd have to remind myself on the exact rules, but returning {"code":
> 200, "json": "stuff"} will send "stuff" through the JSON serializer,
> which means it has to conform to unicode escaping rules that
> mochijson2 exposes via xmerl:ucs16_to_char or whatever.
>
> Perhaps you can log the response you're sending and paste a copy of
> that?
>
> HTH,
> Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Nov 10, 2009 at 5:56 PM, Lennart Melzer <l....@tu-bs.de> wrote:
> Hi Paul,
>
> I'm not trying to return arbitrary binary data,I really just try to post an
> attached file to an external handler. Right now the handler returns a static
> response which works when posting form data (without binary data in it) and
> with get request. It doesn't work when posting a binary file. I don't think
> it's a problem with the handler itself.
>
> I don't know why the couchdb is trying to encode the binary post request as
> json.
>
> Greetings,
>
> Lennart
>
> On Nov 10, 2009, at 11:26 PM, Paul Davis wrote:
>
>> On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l....@tu-bs.de> wrote:
>>>
>>> I just looked through the error logs and the code a bit more. I think
>>> this
>>> passage describes best what seems to happen (though I don't know why)
>>>
>>> So the stracktrace shows that the mochiweb passes the request to
>>> couch_httpd:handle_request where it seems to try to call send_json
>>> (whysoever). It all fails when it tries to encode the binary data as a
>>> json
>>> string. This all seems to happen before couch_httpd_external.erl stuff
>>> gets
>>> called, which might be able to handle binary data correctly.
>>>
>>> Here's the log snippet:
>>> crasher:
>>> initial call: mochiweb_socket_server:acceptor_loop/1
>>> pid: <0.1059.0>
>>> registered_name: []
>>> exception exit: {ucs,{bad_utf8_character_code}}
>>> in function xmerl_ucs:from_utf8/1
>>> in call from mochijson2:json_encode_string/2
>>> in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
>>> in call from lists:foldl/3
>>> in call from mochijson2:json_encode_proplist/2
>>> in call from couch_httpd:send_json/4
>>> in call from couch_httpd:handle_request/5
>>> in call from mochiweb_http:headers/5
>>>
>>> Anyone out that knows more about the internals of couchdb than I do? I
>>> would
>>> be grateful for any advice (gotta learn some erlang programming soonish).
>>>
>>> Greetings,
>>>
>>> Lennart
>>>
>>>
>>> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>>>
>>>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l....@tu-bs.de>
>>>> wrote:
>>>>>
>>>>> I am running Revision 832477 of the couchdb trunk
>>>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>>>
>>>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l....@tu-bs.de>
>>>>>> wrote:
>>>>>>>
>>>>>>> Right now, posting or putting binary data to an external handler
>>>>>>> fails,
>>>>>>> since the data is parsed as a UTF-8 String and not treated as binary.
>>>>>>> The
>>>>>>> external handler never gets to do its work, so this might be an issue
>>>>>>> with
>>>>>>> the handling of data sent to an external handler. Am I misusing the
>>>>>>> external-handler interface?
>>>>>>>
>>>>>>> Error snippet:
>>>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>>>> {<0.95.0>,std_error,
>>>>>>> {mochiweb_socket_server,235,
>>>>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>>>
>>>>>>> Greetings,
>>>>>>>
>>>>>>> Lennart
>>>>>>>
>>>>>>
>>>>>> Lennart,
>>>>>>
>>>>>> What version of CouchDB are you on? I seem to remember a patch for
>>>>>> this going into trunk at some point. Not sure if that was pre/post
>>>>>> 0.10 though.
>>>>>>
>>>>>> Paul Davis
>>>>>
>>>>>
>>>>
>>>> Lennart,
>>>>
>>>> Hmm, looking through the svn log I'm not seeing a commit that mentions
>>>> body or posts. I swear I remember something like that coming through
>>>> at one point. Benoit, didn't you have something to do with that?
>>>>
>>>> Paul Davis
>>>
>>>
>>
>> Lennart,
>>
>> That stack trace suggests that you're trying to return a JSON response
>> that is not valid JSON. If your external is trying to return arbitrary
>> binary data in the _external response object you'll need to make sure
>> and use the body member.
>>
>> Paul Davis
>
>
Lennart,
What are you returning from your _external handler? You should be
returning something like:
{
"code": 200,
"body": "arbitrary string of data and some stuff",
"headers": {"Content-Type": "application/my-stuff"}
}
I'd have to remind myself on the exact rules, but returning {"code":
200, "json": "stuff"} will send "stuff" through the JSON serializer,
which means it has to conform to unicode escaping rules that
mochijson2 exposes via xmerl:ucs16_to_char or whatever.
Perhaps you can log the response you're sending and paste a copy of that?
HTH,
Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Lennart Melzer <l....@tu-bs.de>.
Hi Paul,
I'm not trying to return arbitrary binary data,I really just try to
post an attached file to an external handler. Right now the handler
returns a static response which works when posting form data (without
binary data in it) and with get request. It doesn't work when posting
a binary file. I don't think it's a problem with the handler itself.
I don't know why the couchdb is trying to encode the binary post
request as json.
Greetings,
Lennart
On Nov 10, 2009, at 11:26 PM, Paul Davis wrote:
> On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l....@tu-bs.de>
> wrote:
>> I just looked through the error logs and the code a bit more. I
>> think this
>> passage describes best what seems to happen (though I don't know why)
>>
>> So the stracktrace shows that the mochiweb passes the request to
>> couch_httpd:handle_request where it seems to try to call send_json
>> (whysoever). It all fails when it tries to encode the binary data
>> as a json
>> string. This all seems to happen before couch_httpd_external.erl
>> stuff gets
>> called, which might be able to handle binary data correctly.
>>
>> Here's the log snippet:
>> crasher:
>> initial call: mochiweb_socket_server:acceptor_loop/1
>> pid: <0.1059.0>
>> registered_name: []
>> exception exit: {ucs,{bad_utf8_character_code}}
>> in function xmerl_ucs:from_utf8/1
>> in call from mochijson2:json_encode_string/2
>> in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
>> in call from lists:foldl/3
>> in call from mochijson2:json_encode_proplist/2
>> in call from couch_httpd:send_json/4
>> in call from couch_httpd:handle_request/5
>> in call from mochiweb_http:headers/5
>>
>> Anyone out that knows more about the internals of couchdb than I
>> do? I would
>> be grateful for any advice (gotta learn some erlang programming
>> soonish).
>>
>> Greetings,
>>
>> Lennart
>>
>>
>> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>>
>>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l.melzer@tu-
>>> bs.de> wrote:
>>>>
>>>> I am running Revision 832477 of the couchdb trunk
>>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>>
>>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l.melzer@tu-
>>>>> bs.de>
>>>>> wrote:
>>>>>>
>>>>>> Right now, posting or putting binary data to an external
>>>>>> handler fails,
>>>>>> since the data is parsed as a UTF-8 String and not treated as
>>>>>> binary.
>>>>>> The
>>>>>> external handler never gets to do its work, so this might be an
>>>>>> issue
>>>>>> with
>>>>>> the handling of data sent to an external handler. Am I misusing
>>>>>> the
>>>>>> external-handler interface?
>>>>>>
>>>>>> Error snippet:
>>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>>> {<0.95.0>,std_error,
>>>>>> {mochiweb_socket_server,235,
>>>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>>
>>>>>> Greetings,
>>>>>>
>>>>>> Lennart
>>>>>>
>>>>>
>>>>> Lennart,
>>>>>
>>>>> What version of CouchDB are you on? I seem to remember a patch for
>>>>> this going into trunk at some point. Not sure if that was pre/post
>>>>> 0.10 though.
>>>>>
>>>>> Paul Davis
>>>>
>>>>
>>>
>>> Lennart,
>>>
>>> Hmm, looking through the svn log I'm not seeing a commit that
>>> mentions
>>> body or posts. I swear I remember something like that coming through
>>> at one point. Benoit, didn't you have something to do with that?
>>>
>>> Paul Davis
>>
>>
>
> Lennart,
>
> That stack trace suggests that you're trying to return a JSON response
> that is not valid JSON. If your external is trying to return arbitrary
> binary data in the _external response object you'll need to make sure
> and use the body member.
>
> Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Nov 10, 2009 at 5:22 PM, Lennart Melzer <l....@tu-bs.de> wrote:
> I just looked through the error logs and the code a bit more. I think this
> passage describes best what seems to happen (though I don't know why)
>
> So the stracktrace shows that the mochiweb passes the request to
> couch_httpd:handle_request where it seems to try to call send_json
> (whysoever). It all fails when it tries to encode the binary data as a json
> string. This all seems to happen before couch_httpd_external.erl stuff gets
> called, which might be able to handle binary data correctly.
>
> Here's the log snippet:
> crasher:
> initial call: mochiweb_socket_server:acceptor_loop/1
> pid: <0.1059.0>
> registered_name: []
> exception exit: {ucs,{bad_utf8_character_code}}
> in function xmerl_ucs:from_utf8/1
> in call from mochijson2:json_encode_string/2
> in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
> in call from lists:foldl/3
> in call from mochijson2:json_encode_proplist/2
> in call from couch_httpd:send_json/4
> in call from couch_httpd:handle_request/5
> in call from mochiweb_http:headers/5
>
> Anyone out that knows more about the internals of couchdb than I do? I would
> be grateful for any advice (gotta learn some erlang programming soonish).
>
> Greetings,
>
> Lennart
>
>
> On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
>
>> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l....@tu-bs.de> wrote:
>>>
>>> I am running Revision 832477 of the couchdb trunk
>>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>>
>>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l....@tu-bs.de>
>>>> wrote:
>>>>>
>>>>> Right now, posting or putting binary data to an external handler fails,
>>>>> since the data is parsed as a UTF-8 String and not treated as binary.
>>>>> The
>>>>> external handler never gets to do its work, so this might be an issue
>>>>> with
>>>>> the handling of data sent to an external handler. Am I misusing the
>>>>> external-handler interface?
>>>>>
>>>>> Error snippet:
>>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>>> {<0.95.0>,std_error,
>>>>> {mochiweb_socket_server,235,
>>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>>
>>>>> Greetings,
>>>>>
>>>>> Lennart
>>>>>
>>>>
>>>> Lennart,
>>>>
>>>> What version of CouchDB are you on? I seem to remember a patch for
>>>> this going into trunk at some point. Not sure if that was pre/post
>>>> 0.10 though.
>>>>
>>>> Paul Davis
>>>
>>>
>>
>> Lennart,
>>
>> Hmm, looking through the svn log I'm not seeing a commit that mentions
>> body or posts. I swear I remember something like that coming through
>> at one point. Benoit, didn't you have something to do with that?
>>
>> Paul Davis
>
>
Lennart,
That stack trace suggests that you're trying to return a JSON response
that is not valid JSON. If your external is trying to return arbitrary
binary data in the _external response object you'll need to make sure
and use the body member.
Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Lennart Melzer <l....@tu-bs.de>.
I just looked through the error logs and the code a bit more. I think
this passage describes best what seems to happen (though I don't know
why)
So the stracktrace shows that the mochiweb passes the request to
couch_httpd:handle_request where it seems to try to call send_json
(whysoever). It all fails when it tries to encode the binary data as a
json string. This all seems to happen before couch_httpd_external.erl
stuff gets called, which might be able to handle binary data correctly.
Here's the log snippet:
crasher:
initial call: mochiweb_socket_server:acceptor_loop/1
pid: <0.1059.0>
registered_name: []
exception exit: {ucs,{bad_utf8_character_code}}
in function xmerl_ucs:from_utf8/1
in call from mochijson2:json_encode_string/2
in call from mochijson2:'-json_encode_proplist/2-fun-0-'/3
in call from lists:foldl/3
in call from mochijson2:json_encode_proplist/2
in call from couch_httpd:send_json/4
in call from couch_httpd:handle_request/5
in call from mochiweb_http:headers/5
Anyone out that knows more about the internals of couchdb than I do? I
would be grateful for any advice (gotta learn some erlang programming
soonish).
Greetings,
Lennart
On Nov 10, 2009, at 12:13 AM, Paul Davis wrote:
> On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l....@tu-bs.de>
> wrote:
>> I am running Revision 832477 of the couchdb trunk
>> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>>
>>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l.melzer@tu-
>>> bs.de> wrote:
>>>>
>>>> Right now, posting or putting binary data to an external handler
>>>> fails,
>>>> since the data is parsed as a UTF-8 String and not treated as
>>>> binary. The
>>>> external handler never gets to do its work, so this might be an
>>>> issue
>>>> with
>>>> the handling of data sent to an external handler. Am I misusing the
>>>> external-handler interface?
>>>>
>>>> Error snippet:
>>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>>> {<0.95.0>,std_error,
>>>> {mochiweb_socket_server,235,
>>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>>
>>>> Greetings,
>>>>
>>>> Lennart
>>>>
>>>
>>> Lennart,
>>>
>>> What version of CouchDB are you on? I seem to remember a patch for
>>> this going into trunk at some point. Not sure if that was pre/post
>>> 0.10 though.
>>>
>>> Paul Davis
>>
>>
>
> Lennart,
>
> Hmm, looking through the svn log I'm not seeing a commit that mentions
> body or posts. I swear I remember something like that coming through
> at one point. Benoit, didn't you have something to do with that?
>
> Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Nov 9, 2009 at 12:15 PM, Lennart Melzer <l....@tu-bs.de> wrote:
> I am running Revision 832477 of the couchdb trunk
> On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
>
>> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l....@tu-bs.de> wrote:
>>>
>>> Right now, posting or putting binary data to an external handler fails,
>>> since the data is parsed as a UTF-8 String and not treated as binary. The
>>> external handler never gets to do its work, so this might be an issue
>>> with
>>> the handling of data sent to an external handler. Am I misusing the
>>> external-handler interface?
>>>
>>> Error snippet:
>>> [error] [<0.95.0>] {error_report,<0.30.0>,
>>> {<0.95.0>,std_error,
>>> {mochiweb_socket_server,235,
>>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>>
>>> Greetings,
>>>
>>> Lennart
>>>
>>
>> Lennart,
>>
>> What version of CouchDB are you on? I seem to remember a patch for
>> this going into trunk at some point. Not sure if that was pre/post
>> 0.10 though.
>>
>> Paul Davis
>
>
Lennart,
Hmm, looking through the svn log I'm not seeing a commit that mentions
body or posts. I swear I remember something like that coming through
at one point. Benoit, didn't you have something to do with that?
Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Lennart Melzer <l....@tu-bs.de>.
I am running Revision 832477 of the couchdb trunk
On Nov 9, 2009, at 6:09 PM, Paul Davis wrote:
> On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l....@tu-bs.de>
> wrote:
>> Right now, posting or putting binary data to an external handler
>> fails,
>> since the data is parsed as a UTF-8 String and not treated as
>> binary. The
>> external handler never gets to do its work, so this might be an
>> issue with
>> the handling of data sent to an external handler. Am I misusing the
>> external-handler interface?
>>
>> Error snippet:
>> [error] [<0.95.0>] {error_report,<0.30.0>,
>> {<0.95.0>,std_error,
>> {mochiweb_socket_server,235,
>> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>>
>> Greetings,
>>
>> Lennart
>>
>
> Lennart,
>
> What version of CouchDB are you on? I seem to remember a patch for
> this going into trunk at some point. Not sure if that was pre/post
> 0.10 though.
>
> Paul Davis
Re: POST/PUTing binary Attachments to external Handler
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Nov 9, 2009 at 11:54 AM, Lennart Melzer <l....@tu-bs.de> wrote:
> Right now, posting or putting binary data to an external handler fails,
> since the data is parsed as a UTF-8 String and not treated as binary. The
> external handler never gets to do its work, so this might be an issue with
> the handling of data sent to an external handler. Am I misusing the
> external-handler interface?
>
> Error snippet:
> [error] [<0.95.0>] {error_report,<0.30.0>,
> {<0.95.0>,std_error,
> {mochiweb_socket_server,235,
> {child_error,{ucs,{bad_utf8_character_code}}}}}}
>
> Greetings,
>
> Lennart
>
Lennart,
What version of CouchDB are you on? I seem to remember a patch for
this going into trunk at some point. Not sure if that was pre/post
0.10 though.
Paul Davis