You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by "Joan Touzet (JIRA)" <ji...@apache.org> on 2009/05/07 00:15:30 UTC
[jira] Created: (COUCHDB-345) High ASCII can be
High ASCII can be
------------------
Key: COUCHDB-345
URL: https://issues.apache.org/jira/browse/COUCHDB-345
Project: CouchDB
Issue Type: Bug
Affects Versions: 0.9
Environment: OSX 10.5.6
Reporter: Joan Touzet
It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
================================================
#!/usr/bin/python
from Couch import Couch
db = Couch('localhost', '5984')
db.createDb('utf8_fail')
badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
doc = """
{
"Message":\"""" + badtext + """\",
}
"""
db.saveDoc('utf8_fail', doc, 'fail')
db.openDoc('utf8_fail', 'fail')
================================================
Sample output against 0.9.0 is as follows:
{
"ok": true
}
{
"id": "fail",
"ok": true,
"rev": "1-76726372"
}
{
"error": "ucs",
"reason": "{bad_utf8_character_code}"
}
================================================
Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Curt Arnold updated COUCHDB-345:
--------------------------------
Attachment: enctest.zip
This is a JUnit 4 test case (with corresponding pom.xml) that demonstrates the current broken behavior (or at least of about a week or so ago).
Documents that are not valid UTF-8 are accepted into the database, but can not be retrieved. I did not test if they broken queries, but have no reason to doubt that misencoded documents would cause unexpected behavior in the database. It would seem plausible that an attacker could seriously damage a CouchDB application by inserting misencoded documents. Depending on an API layer to not send misencoded documents would still leave the DB vulnerable to an intentional attack or a miscoded API layer.
The test creates http://localhost:5984/testdb and then tries to insert 5 documents. The first is just straight ASCII, the second inserts a document containing \u00C0 - \u00C6 encoded in UTF-8 and the 3rd inserts the same document, but with the characters escaped instead of UTF-8 encoded. These three behavior as expected.
The next two attempt to insert the same characters, but instead of UTF-8 encoded, they are ISO-8859-1 encoded (that is the byte sequence 0xC0, 0xC1, 0xC2 ... is in the body). One attempt is with an Content-Encoding=ISO-8559-1 and the other without. Both PUT returns with an 201 response, but an attempt to fetch results in a 500 due with an encoding error stack trace. Returning a 400 in both cases would be appropriate since the RFC says that JSON is always UTF-8 encoded.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749156#action_12749156 ]
Jan Lehnardt commented on COUCHDB-345:
--------------------------------------
the unicode module is new in R13.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749207#action_12749207 ]
Adam Kocoloski commented on COUCHDB-345:
----------------------------------------
Hi Curt, thanks for all your work on this ticket. I did a bit of simple benchmarking using code from the eep0018 project, which indicated that couch_db:json_decode increased decoding time by about 12% over trunk. The patch I submitted added about 4%. I wouldn't put much stock in the difference between these numbers, but it does support your expectation that xmerl_ucs:from_utf8 is fast compared to the actual JSON decoding.
I'm comfortable with either patch. I'm not so concerned about diverging from stock MochiWeb. We already diverge in order to correctly handle escaped UTF-16 surrogate pairs, and to use the generally-agreed-upon ejson term format instead of the {obj, Data} one in the stock distribution. We submitted a patch for the surrogate pair issue quite a while ago, but it hasn't been applied. We could submit a patch for this, too, as rejecting bad input is a feature that any JSON decoder should have.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Joan Touzet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joan Touzet updated COUCHDB-345:
--------------------------------
Description:
It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
Sample output against 0.9.0 is as follows:
================================================
{
"ok": true
}
{
"id": "fail",
"ok": true,
"rev": "1-76726372"
}
{
"error": "ucs",
"reason": "{bad_utf8_character_code}"
}
================================================
Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
was:
It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
================================================
#!/usr/bin/python
from Couch import Couch
db = Couch('localhost', '5984')
db.createDb('utf8_fail')
badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
doc = """
{
"Message":\"""" + badtext + """\",
}
"""
db.saveDoc('utf8_fail', doc, 'fail')
db.openDoc('utf8_fail', 'fail')
================================================
Sample output against 0.9.0 is as follows:
{
"ok": true
}
{
"id": "fail",
"ok": true,
"rev": "1-76726372"
}
{
"error": "ucs",
"reason": "{bad_utf8_character_code}"
}
================================================
Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Noah Slater (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749062#action_12749062 ]
Noah Slater commented on COUCHDB-345:
-------------------------------------
It is unpossible to have Unicode strings. That breaks science.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749160#action_12749160 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
I looked at the Erlang test suite and didn't see anything that looked similar to what I wanted to test. When I did "make check", I got a lot of lines like:
"test/etap/070-couch-db....................dubious"
Test returned status 127 (wstat 32512, 0x7f00)
Looks like it is beyond my skills at the moment to write an Erlang test that would effectively test the changes.
If you wanted to accept interpret badly encoded UTF-8 as ISO-8859-1, I'd suggest logging it, but in the catch, you could do something like:
mochijson2:decode(xmerl_ucs:to_utf8(binary_to_list(S))
Sniffing the multi-byte encodings could also be done in couch_db:json_decode as an enhancement. You don't have the Content-Encoding header available, so you could not support users who explicitly try to send JSON in arbitrary encodings. However, since JSON is supposed to be encoded in Unicode and you can unambiguously determine the encoding from the content. I think it is best to ignore the Content-Encoding even if it was available.
I do not know the performance implications of the patch. It would likely be cheaper if the binary could be scanned only and not extracted to a list, but I think that has to be relatively minor compared to the JSON parsing.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758726#action_12758726 ]
Jan Lehnardt commented on COUCHDB-345:
--------------------------------------
I'd like to defer decision on this patch until after 0.10. I don't think this should block 0.10 and we can fix this in 0.10.x (x > 0) under the following conditions: the 0.10.0 release notes contain a link to this issue and explain that the current acceptance of non-utf8 input is discouraged and should not be relied upon as it will be removed in later releases.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749058#action_12749058 ]
Paul Joseph Davis commented on COUCHDB-345:
-------------------------------------------
Thanks for tracking this down to find a repeatable test. Couple notes:
I actually got caught by this just the other day, but the json rfc says that strings must be Unicode, not utf-8. Ie, utf-8, 16 and 32, and both BE and LE for 16 and 32. I don't think I've ever seen a completely compliant parser.
Does anyone know if this byte range is special? I'm no encoding expert so I don't have a good feel for what might be the issue.
While I appreciate your tracking this down, in the future could you submit test cases as JavaScript, erlang, or a bash script? Not many of us have a java dev environment setup to run junit tests.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Noah Slater (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749090#action_12749090 ]
Noah Slater commented on COUCHDB-345:
-------------------------------------
Forcing the request body to be UTF-8 is lazy and inconsiderate.
CouchDB should attempt to decode any request body according to the Content-Encoding header.
If no such header exists, CouchDB should do one of the following:
* attempt to decode as UTF-8
* attempt to guess the character encoding, and decode
If either of these three methods fail, the request body is garbage and should be rejected with a 4xx client error.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski updated COUCHDB-345:
-----------------------------------
Attachment: reject_invalid_utf8.patch
Hi folks, here's a patch that rejects updates which are not valid UTF-8. I tested it against Joan's and Mark's Python scripts, but I don't have junit so I wasn't able to run Curt's test.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706689#action_12706689 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
Not a CouchDB or Python developer, but interested in the issue and thought I might help you clarify what you have encountered.
According to http://www.ietf.org/rfc/rfc4627.txt?number=4627, JSON data must be encoded in Unicode and the default encoding is UTF-8. I'm assuming that is the definite statement on JSON encoding.
It should not be necessary to escape non US-ASCII characters as long as the document is properly encoded in UTF-8, one of the UTF-16s or UCS-4. What I'm guessing is happening in your code fragment is that a Latin-1 (ISO-8859-1) byte sequence is sent in the request which should have been rejected by the PUT operation since it bad JSON.
If that interpretation is correct and shooting from the hip, I think:
1. The Python API should have either rejected the standard string or converted it to a Unicode string before sending the request.
2. CouchDB should reject any document that is not valid UTF-8, UTF-16BE, UTF-16LE or UCS-4.
3. It should not require non-USASCII characters to be escaped.
4. CouchDB should ignore any encoding specified in the header. Limiting to only Unicode types means that the encoding can be reliably determined and it is likely that any encoding in the header might be wrong.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749162#action_12749162 ]
Jan Lehnardt commented on COUCHDB-345:
--------------------------------------
Hi Curt, to run `make check` successfully, you need to install etap first: http://github.com/ngerakines/etap/
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749085#action_12749085 ]
Adam Kocoloski commented on COUCHDB-345:
----------------------------------------
oh, and the other funny thing is that with this patch applied, the oauth and cookie_auth tests break, because they try to upload a secret that is invalid UTF-8 :-P.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Adam Kocoloski closed COUCHDB-345.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.10
Assignee: Adam Kocoloski
Ok, I've committed the patch I wrote to trunk and the 0.10 release branch. I'll submit it upstream to the Mochi team as well. We still need to address the issue of alternate encodings and Content-Encoding headers so we don't come across as "lazy and inconsiderate" :)
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Assignee: Adam Kocoloski
> Fix For: 0.10
>
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758729#action_12758729 ]
Adam Kocoloski commented on COUCHDB-345:
----------------------------------------
I'm on board with that. I don't have the time to properly benchmark the different patches at the moment, and if we're serious about patching mochijson2 to reject bad input we should get their thoughts on the matter as well.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Noah Slater (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749164#action_12749164 ]
Noah Slater commented on COUCHDB-345:
-------------------------------------
I disagree with Curt.
The JSON RFC is either wrong or carelessly worded. You cannot encode anything as Unicode because Unicode is not an encoding, it is a collection of code points that have no binary representation. You can encode these code points into character data, and you can decode the same character data into Unicode. Unicode is always some internal representation after decoding, and before encoding. I am guessing everyone already knows this, but I keep seeing people form arguments (particularly on IRC) that start with "since JSON has to be encoded as Unicode" which is just a meaningless sentence (and the RFC is to blame as it uses this wording) and hence conclusions that follow from that have tended to be false.
In this case, what the JSON RFC should say is that JSON should be encoded from Unicode, which means that the encoding could be anything from ISO-8859-1 to Shift JIS, which means that we cannot "unambiguously determine the encoding from the content." Even if we decided to only allow UTF-8, UTF-16, or UTF-32, we could only "unambiguously determine the encoding" if the request body included the BOM, which is entirely optional. So again, without the Content-Encoding information, we are forced to use a heuristic. Heuristics already exist, and where they are not already available in Erlang, I rather suspect that they can be ported with relative ease.
If we can access the Content-Encoding, we should absolutely use it, and absolutely reject as garbage any request that could not be decoded with the explicit encoding. Any patch that will-fully ignored this information only to fall back onto a heuristic would get my emphatic veto. I am however satisfied with requiring UTF-8 in the short term, and adding Content-Encoding awareness at some later point.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758844#action_12758844 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
>I'd like to defer decision on this patch until after 0.10. I don't think this should block 0.10 and we can fix this in 0.10.x (x > 0) under the following conditions: >the 0.10.0 release notes contain a link to this issue and explain that the current acceptance of non-utf8 input is discouraged and should not be relied >upon as it will be removed in later releases.
The only people who would be intentionally inserting malformed documents would be those who are intentionally trying to crash a database. Are you saying that people who are trying to attack a deployed CouchDB instance would be discouraged from using this attack since it might go away in the future?
Without either of the patches, any CouchDB instance is vulnerable to serious degradation or failure from anyone who can put a document in the database.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Paul Joseph Davis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758738#action_12758738 ]
Paul Joseph Davis commented on COUCHDB-345:
-------------------------------------------
I'd vote for applying Adam's patch to reject the utf-8 as a stop gap. Then when we become more enterprising we should patch mochiweb_request to mux the incoming body to utf-8. For the short term rejecting non-utf8 seems more sane than accepting it and rejecting to return the stored data.
On a side note I see a note about xmerl_ucs:from_utf8 and friends. There's a unicode:is_7bit_clean (or similarly named) function that might be useful.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Curt Arnold updated COUCHDB-345:
--------------------------------
Attachment: badenc1.patch
The patch also addresses the issue. It redefined ?JSON_DECODE to couch_db:json_decode/1 which calls xmerl_ucs:from_utf8 prior to delegating to mochijson2:decode. If there is a encoding problem, it throws {invalid_json, <<"bad utf8...">> } which eliminates the need to add new catch clauses in the calling code.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Joan Touzet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joan Touzet updated COUCHDB-345:
--------------------------------
Description:
It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
================================================
#!/usr/bin/python
from Couch import Couch
db = Couch('localhost', '5984')
db.createDb('utf8_fail')
badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
doc = """
{
"Message":\"""" + badtext + """\",
}
"""
db.saveDoc('utf8_fail', doc, 'fail')
db.openDoc('utf8_fail', 'fail')
================================================
Sample output against 0.9.0 is as follows:
{
"ok": true
}
{
"id": "fail",
"ok": true,
"rev": "1-76726372"
}
{
"error": "ucs",
"reason": "{bad_utf8_character_code}"
}
================================================
Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
was:
It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
================================================
#!/usr/bin/python
from Couch import Couch
db = Couch('localhost', '5984')
db.createDb('utf8_fail')
badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
doc = """
{
"Message":\"""" + badtext + """\",
}
"""
db.saveDoc('utf8_fail', doc, 'fail')
db.openDoc('utf8_fail', 'fail')
================================================
Sample output against 0.9.0 is as follows:
{
"ok": true
}
{
"id": "fail",
"ok": true,
"rev": "1-76726372"
}
{
"error": "ucs",
"reason": "{bad_utf8_character_code}"
}
================================================
Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
Summary: "High ASCII" can be inserted into db but not retrieved (was: High ASCII can be )
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
> ================================================
> #!/usr/bin/python
> from Couch import Couch
> db = Couch('localhost', '5984')
> db.createDb('utf8_fail')
> badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
> doc = """
> {
> "Message":\"""" + badtext + """\",
> }
> """
> db.saveDoc('utf8_fail', doc, 'fail')
> db.openDoc('utf8_fail', 'fail')
> ================================================
> Sample output against 0.9.0 is as follows:
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749149#action_12749149 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
The patch does result in passing the tests as they were intended to be (asserting that the PUT returns a 400). The versions that were attached were demonstrating the failure of the GET after the PUT so they didn't assert the 400 return. The patch hunk for couch_httpd.erl is a little stale and needs to be manually applied.
The patch modifies mochijson2, so it puts us in the position of diverging from stock MochiWeb.
My thought was to put a call to unicode:characters_to_binary(Bin,utf8,utf8) in PUT code path. If the source Bin is valid UTF-8, the return value will be identical. If not, then it returns { error, "Valid characters", <<MalformedStuff>> }. Support for the UTF-16's could be done at the same place.
http://erlang.org/doc/apps/stdlib/unicode_usage.html and http://erlang.org/doc/man/unicode.html mention that the implementation is complete as documented in R13A, but I don't know how much if any of the unicode module is present in R12B5. Mochiweb references xmerl_ucs, which isn't in the docs but is apparently the ucs string support for the XML parser.
I'd suggest implementing a check/conversion on the PUT code path using the unicode module and then adapting it to run on our minimum platform if that is an issue.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749216#action_12749216 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
I'm guessing that the performance difference would be reduced if couch_db:json_decode were modified so it only called xmerl_ucs:from_utf8 when it encountered a byte value > 0x7F in the binary. I've leave to Adam to code and benchmark it to avoid demonstrating my Erlang newbieness.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758741#action_12758741 ]
Adam Kocoloski commented on COUCHDB-345:
----------------------------------------
I think we wanted to keep R12B-5 as the minimum Erlang version for 0.10, so that means no unicode module.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badenc1.patch, badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COUCHDB-345) "High ASCII" can be inserted into db
but not retrieved
Posted by "Joan Touzet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joan Touzet updated COUCHDB-345:
--------------------------------
Attachment: badtext.tar.gz
Code to reproduce the defect including the pre-requisite wrapper module.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> This sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string. It requires the file Couch.py in the same directory, containing the code from the "Example wrapper class" at http://wiki.apache.org/couchdb/Getting_started_with_Python to run.
> ================================================
> #!/usr/bin/python
> from Couch import Couch
> db = Couch('localhost', '5984')
> db.createDb('utf8_fail')
> badtext = "4E45494D454E2046D85252204641454E21".decode("hex")
> doc = """
> {
> "Message":\"""" + badtext + """\",
> }
> """
> db.saveDoc('utf8_fail', doc, 'fail')
> db.openDoc('utf8_fail', 'fail')
> ================================================
> Sample output against 0.9.0 is as follows:
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Adam Kocoloski (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749108#action_12749108 ]
Adam Kocoloski commented on COUCHDB-345:
----------------------------------------
I agree with both Noah and Jan. Applying this patch is better than not applying it.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Curt Arnold (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749083#action_12749083 ]
Curt Arnold commented on COUCHDB-345:
-------------------------------------
ISO-8859-1, Cp1252 and Latin-1 are near synonyms for encoding the first 256 character points in Unicode as single byte values and is incapable of representing any other character without some escape mechanism. Any arbitrary set of bytes would be a valid ISO-8859-1 sequence and can be decoded into a sequence of Unicode characters.
UTF-8 is an variable byte encoding of the full Unicode character repertoire. Character values from \u0000 to \u007F are represented as a single-byte, while other characters require 2-6 bytes to encode. Unlike ISO-8859-1, not every sequence of bytes is valid and can be converted back to Unicode character points. If I remember correctly, any two-bytes in a row with the high-bit set is invalid. The test date in the last two cases are valid ISO-8859-1 sequence, but they can not be interpreted as UTF-8 since they contain byte sequences that can not be converted back into Unicode code points.
If it was just an encoding mismatch and the data was being misinterpreted, you would lay the blame at the client. However, in this case, data can go into the database that the rest of the stack can't process since it contains invalid sequences.
The RFC mentions the two variants of UTF-16 and UCS-4, however the ISO-8859-1 sequences could not be interpreted using any of those encodings since the first two characters must be ASCII. There are only certain sequences of bytes that could appear for JSON encoded in any of those encodings and the byte sequences send in the last two cases don't match any of those patterns. Sniffing the encoding would work in a similar manner to XML which is described in http://www.w3.org/TR/REC-xml/#sec-guessing.
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COUCHDB-345) "High ASCII" can be inserted into
db but not retrieved
Posted by "Jan Lehnardt (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COUCHDB-345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749103#action_12749103 ]
Jan Lehnardt commented on COUCHDB-345:
--------------------------------------
I'd be happy to see in code what Noah suggests down the road. But until we get such a patch we should go with denying non-utf-8 encoding if we can't handle it correctly. I don't think this is lazy as a first approach
> "High ASCII" can be inserted into db but not retrieved
> ------------------------------------------------------
>
> Key: COUCHDB-345
> URL: https://issues.apache.org/jira/browse/COUCHDB-345
> Project: CouchDB
> Issue Type: Bug
> Affects Versions: 0.9
> Environment: OSX 10.5.6
> Reporter: Joan Touzet
> Attachments: badtext.tar.gz, enctest.zip, reject_invalid_utf8.patch
>
>
> It is possible to PUT/POST a document into CouchDB with a "high ASCII" value that cannot be retrieved. This results from not escaping a non-ASCII value into \u#### when PUT/POSTing the document.
> The attached sample code will recreate the problem using the hex value D8 (Ø) in a possibly unsavoury test string.
> Sample output against 0.9.0 is as follows:
> ================================================
> {
> "ok": true
> }
> {
> "id": "fail",
> "ok": true,
> "rev": "1-76726372"
> }
> {
> "error": "ucs",
> "reason": "{bad_utf8_character_code}"
> }
> ================================================
> Please note this defect turned up another problem, namely that the bad_utf8_character_code exception thrown by a design document attempting to map() the bad document caused Futon to fail silently in building the view, with no indication (except via debug log) that there was a failure. The log indicated two attempts to build the view, both failing, followed by an uncaught exception error for Futon.
> Based on this, there are likely other areas in the codebase that do not handle the bad_utf8_character_code exception correctly.
> My belief is that CouchDB shouldn't accept this input and should have rejected the PUT/POST, or should have escaped the input itself before the insertion.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.