You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Curt Arnold <ca...@apache.org> on 2009/08/30 05:51:51 UTC

Character encodings and JSON RFC (spun off from COUCHDB-345)

I'm spinning a discussion that was occurring in COUCHDB-345 (http://issues.apache.org/jira/browse/COUCHDB-345 
) over to the mailing list since it was growing beyond the immediate  
issue reported.  The reported problem was that CouchDB would accept  
PUT requests without checking that the content contained valid UTF-8  
encoded data which would result in documents that could not be  
retrieved, would disrupt view generation and potentially have other  
adverse side-effects.

Noah Slater added a comment - 29/Aug/09 09:40 AM
> I disagree with Curt.
>
> The JSON RFC is either wrong or carelessly worded. You cannot encode  
> anything as Unicode because Unicode is not an encoding, it is a  
> collection of code points that have no binary representation. You  
> can encode these code points into character data, and you can decode  
> the same character data into Unicode. Unicode is always some  
> internal representation after decoding, and before encoding. I am  
> guessing everyone already knows this, but I keep seeing people form  
> arguments (particularly on IRC) that start with "since JSON has to  
> be encoded as Unicode" which is just a meaningless sentence (and the  
> RFC is to blame as it uses this wording) and hence conclusions that  
> follow from that have tended to be false.


I agree that is unfortunately worded.  I checked the IETF RFC errata  
page (http://www.rfc-editor.org/errata_search.php?rfc=4627) and did  
not find a clarification on this issue.  I basically interpreted "in  
Unicode" as "in a Unicode Transformation Format" and more specifically  
as in a UTF recommended by the Unicode consortium and in widespread use.


There is this quote from http://www.json.org/fatfree.html:
>
> The character encoding of JSON text is always Unicode. UTF-8 is the  
> only encoding that makes sense on the wire, but UTF-16 and UTF-32  
> are also permitted.

It still uses the troublesome meme "character encoding ... Unicode",  
however it seems to be a stretch to read that and think that Shift- 
JIS, ISO-8559-8, MacLatin, EBCDIC, etc are also fine and dandy.

Don Box also seems to have a similar interpretation (http://www.pluralsight.com/community/blogs/dbox/archive/2007/01/03/45560.aspx 
):
>
> 2. Like Joe Gregorio states in the comments on Tim's post, I also  
> prefer JSON's simplification of only allowing UTF-*-based encoding.

Tim Bray in http://www.tbray.org/ongoing/When/200x/2006/12/21/JSON had  
an interesting comment in:
>
> I look at the Ruby JSON library, for example, and I see all these  
> character encoding conversion routines; blecch.
> Use JSON · Seems easy to me; if you want to serialize a data  
> structure that’s not too text-heavy and all you want is for the  
> receiver to get the same data structure with minimal effort, and you  
> trust the other end to get the i18n right, JSON is hunky-dory.

This hints things in the field aren't all pristine UTF encodings  
however.

Probably best to ping the RFC editor to see if there is a clarification.

>
> In this case, what the JSON RFC should say is that JSON should be  
> encoded from Unicode, which means that the encoding could be  
> anything from ISO-8859-1 to Shift JIS, which means that we cannot  
> "unambiguously determine the encoding from the content." Even if we  
> decided to only allow UTF-8, UTF-16, or UTF-32, we could only  
> "unambiguously determine the encoding" if the request body included  
> the BOM, which is entirely optional. So again, without the Content- 
> Encoding information, we are forced to use a heuristic. Heuristics  
> already exist, and where they are not already available in Erlang, I  
> rather suspect that they can be ported with relative ease.

I haven't worked through all the sequences, but since we know that the  
first characters of a JSON string is either " " or "{", that should be  
enough to unambiguously determine whether the only possible encoding  
of the set UTF-8, UTF-16BE, UTF-16LE, or UTF-32.


>
> If we can access the Content-Encoding, we should absolutely use it,  
> and absolutely reject as garbage any request that could not be  
> decoded with the explicit encoding. Any patch that will-fully  
> ignored this information only to fall back onto a heuristic would  
> get my emphatic veto. I am however satisfied with requiring UTF-8 in  
> the short term, and adding Content-Encoding awareness at some later  
> point.

The RFC says "A JSON parser MAY accept non-JSON forms or extensions."   
So unlike an XML processor that is prohibited from assuming ISO-8859-1  
if it encounters a invalid UTF-8 sequence, a JSON parser could  
transparently assume ISO-8859-1 after encountering a bad UTF-8  
sequence.  Whether that would be a good thing  is debatable.

There are a couple of questions that could be addressed:

1. How to treat a JSON request entity that does not contain a Content- 
Encoding header.  Particularly when the entity is not consistent with  
the expected encoding.

2. How to treat a JSON request with a specified Content-Encoding.   
What encodings would be supported?  What would CouchDB do for an  
unsupported encoding?  What would occur if the entity was not  
consistent with the encoding?

3. What should CouchDB send when there is no "Accept-Charset" in the  
request.

4. What should CouchDB send where there is an "Accept-Charset" in the  
request.  Particularly if the request does not contain a UTF.

I think the current answers are:

1. Entity is interpreted as UTF-8.  Currently if the encoding is  
inconsistent, it is still committed to the database and bad things  
happen later.  If a fix for COUCHDB-345 is committed, then CouchDB  
would reject the request with a 400.

2. Same as 1, Content-Encoding is not considered.

3. CouchDB always sends UTF-8.

4. Same as 3, Accept-Charset is not considered.


It is not a pressing issue for me and since COUCHDB-345 languished for  
such a long time, I'm not thinking that many people are trying to push  
other encodings into the database with the exceptions of people  
pushing ISO-8859-1 up but not getting burned since their content  
hasn't yet contained non ASCII characters.


Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Curt Arnold <ca...@apache.org>.
Does anyone have observations from logs et al of what clients are  
currently sending in the way of Content-Encoding and Accept-Charset  
headers?


Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Randall Leeds <ra...@gmail.com>.
On Sun, Aug 30, 2009 at 12:19, Jan Lehnardt <ja...@apache.org> wrote:

> Noah, Paul, this made my morning. Thanks!
>
> Jan
> --


+1

Not only the endianness of British gentlemen, but this entire thread is
hilarious (and also sort of interesting/educational). But then, I'm on
vacation and I think this is fun to read... so maybe I'm not well.

Randall

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Jan Lehnardt <ja...@apache.org>.
On 30 Aug 2009, at 07:49, Noah Slater wrote:

> On Sun, Aug 30, 2009 at 01:46:26AM -0400, Paul Davis wrote:
>> No! Since it was transfered across a network, obviously we can only
>> accept the big-endian variants.
>
> I'll have you know that all British gentlemen are big-endian.
>
> -- 
> Noah Slater, http://tumbolia.org/nslater
>


Noah, Paul, this made my morning. Thanks!

Jan
--



Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 01:46:26AM -0400, Paul Davis wrote:
> No! Since it was transfered across a network, obviously we can only
> accept the big-endian variants.

I'll have you know that all British gentlemen are big-endian.

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
On Sun, Aug 30, 2009 at 1:42 AM, Noah Slater<ns...@apache.org> wrote:
> On Sun, Aug 30, 2009 at 01:27:18AM -0400, Paul Davis wrote:
>> I've never seen a self contained JSON parser that is compliant with
>> anything other than UTF-8. You could argue that Python's is, but it
>> forces all input to it's internal Unicode representation AFAIK.
>
> Yeah, well, software sucks. Init?
>
>> > I vote for the first option.
>>
>> Patches welcome. :)
>
> I prefer being a Unicode snob on the mailing lists, kthx.
>
>> I thought you said on IRC that the RFC's detection scheme only works
>> if the BOM is specified which is non mandatory. If it's not mandatory
>> then it'd be a guess. Even if the major encodings can be determined
>> I'd invent an encoding spec just to prove its still a guess.
>
> The JSON RFC has a fool proof method that doesn't involve the BOM.
>
> I quoted this earlier in the thread.
>

And I thought I quoted you quoting something about quotes when I said
you said that foolproof method was about as fool proof as a wet cloth.
Granted I could've mis intrepreted. Curt had a link to the XML style
detection which also seemed to "when clients do things that clients do
you have to break down and try other stuff" and the end result was
"maybe possibly utf-8 but not certain".

>> UTF-8 obviously. For 16 and 32 we can obviously only accept BE
>> variants since it was sent via HTTP.
>
> I hope you didn't just use BE to mean British English.

No! Since it was transfered across a network, obviously we can only
accept the big-endian variants.

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 01:27:18AM -0400, Paul Davis wrote:
> I've never seen a self contained JSON parser that is compliant with
> anything other than UTF-8. You could argue that Python's is, but it
> forces all input to it's internal Unicode representation AFAIK.

Yeah, well, software sucks. Init?

> > I vote for the first option.
>
> Patches welcome. :)

I prefer being a Unicode snob on the mailing lists, kthx.

> I thought you said on IRC that the RFC's detection scheme only works
> if the BOM is specified which is non mandatory. If it's not mandatory
> then it'd be a guess. Even if the major encodings can be determined
> I'd invent an encoding spec just to prove its still a guess.

The JSON RFC has a fool proof method that doesn't involve the BOM.

I quoted this earlier in the thread.

> > If it is explicit, and it is wrong, vomit in their face.
>
> FOUURRRROHHHSIXXXXX. Oh, pardon me. Late night last night.

Zing!

> >> > What encodings would be supported?
> >>
> >> Patches welcome. UTF-8 currently kinda sort supported.
> >
> > UTF-8, UTF-16, and UTF-36
>
> UTF-8 obviously. For 16 and 32 we can obviously only accept BE
> variants since it was sent via HTTP.

I hope you didn't just use BE to mean British English.

> > I would prefer vomiting, but any kind of humiliation works for me.
>
> lol. More awesome quotes plzkthx.

I woke up in a ditch, once. True story.

Best,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
>  Can UTF-8 represent all possible Unicode code points?
>
> To which I would have replied "yes."

You got it.

> Any conformant JSON parser should accept UTF-8, UTF-16, and UTF-36. I mean, it's
>
> right there in the spec as a requirement, along with rocking horse people,
> eating marshmelow pies.

I've never seen a self contained JSON parser that is compliant with
anything other than UTF-8. You could argue that Python's is, but it
forces all input to it's internal Unicode representation AFAIK.

> If we support JSON, we either:
>
>  * Support UTF-8, UTF-16, and UTF-36 per the insane spec.
>
>  * Willfully ignore the spec and require UTF-8.
>
> I vote for the first option.

Patches welcome. :)

>> Assume UTF-8. If fail, maybe try guessing. If fail, throw a meatball
>> at the client saying rejected. We already ignore quite a few headers
>> and do things "Non-RESTful-ly" so I'm not too concerned.
>
> No need to guess. The RFC has a mechanism for determining the encoding. It also
> specifies a way to indicate the encoding. Which is totally not redundant, in any
> way. Thankfully, it doesn't tell us how to handle conflicting information in
> this respect, because of course, that would be too easy. Software is no fun when
> the specifications make things obvious.

I thought you said on IRC that the RFC's detection scheme only works
if the BOM is specified which is non mandatory. If it's not mandatory
then it'd be a guess. Even if the major encodings can be determined
I'd invent an encoding spec just to prove its still a guess.

>> > 2. How to treat a JSON request with a specified Content-Encoding.
>>
>> If the encoding is understood, transcode to a UTF-8 representation.
>
> If it is explicit, and it is wrong, vomit in their face.

FOUURRRROHHHSIXXXXX. Oh, pardon me. Late night last night.

>> > What encodings would be supported?
>>
>> Patches welcome. UTF-8 currently kinda sort supported.
>
> UTF-8, UTF-16, and UTF-36

UTF-8 obviously. For 16 and 32 we can obviously only accept BE
variants since it was sent via HTTP.

> I would prefer vomiting, but any kind of humiliation works for me.

lol. More awesome quotes plzkthx.

HTH,
Paul Davis

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 12:33:19AM -0400, Paul Davis wrote:
> Can UTF-8 represent all possible unicode encodings?

I could reinterpret this nonsense as:

  Can UTF-8 represent all possible Unicode code points?

To which I would have replied "yes."

> I'm gonna assume yes for the rest of this post.

Good-o.

> I think we're in a bit of a weird spot here cause we're playing with
> the head butt of two different RFC's. The HTTP transport RFC that
> deals with Content-Encoding and charset awesomeness and the JSON RFC
> that is so full of ambiguity I'd like to kick it. On the plus side,
> there's so much ambiguity here that we can basically do whatever we
> want and no one can accuse us of being wrong.

Actually, JSON spells out what we MUST do here:

  JSON may be represented using UTF-8, UTF-16, or UTF-32.  When JSON
  is written in UTF-8, JSON is 8bit compatible.  When JSON is
  written in UTF-16 or UTF-32, the binary content-transfer-encoding
  must be used.

To be honest though, I think that using a content-transfer-encoding header is
further proof that the RFC editor was high. I mean, seriously, this doesn't even
make any sense. JSON can be UTF-8, UTF-16, or UTF-32 in a non-network context,
but if you send it over the wire, it's suddenly a transfer-encoding? It almost
sounds like they were playing technical jargon bingo.

Unicode! Transfer Encoding! Do I win a prize?

> That said, I think we should isolate concerns. Unless someone want's
> to write a JSON parser that understands multiple character encodings
> and doesn't suck ass performance wise, we should probably just assume
> the JSON parser is UTF-8 only.

Any conformant JSON parser should accept UTF-8, UTF-16, and UTF-36. I mean, it's

right there in the spec as a requirement, along with rocking horse people,
eating marshmelow pies.

> Before anyone goes hollering about that, we still have the HTTP layer
> to play with in terms of accepting content encoding. And nothing in
> the HTTP layer says we have to accept UC-4 or NR-17 or whatever. So
> while we're more than welcome to reject any request bodies way before
> they hit the JSON serializer, Noah would probably cut my throat for
> suggesting we don't play nice. Either way, this big conversation on
> character encodings should probably focus on how we move things to
> UTF-8 which I officially nominate as the already de-facto CouchDB
> character encoding.

If we support JSON, we either:

  * Support UTF-8, UTF-16, and UTF-36 per the insane spec.

  * Willfully ignore the spec and require UTF-8.

I vote for the first option.

> Assume UTF-8. If fail, maybe try guessing. If fail, throw a meatball
> at the client saying rejected. We already ignore quite a few headers
> and do things "Non-RESTful-ly" so I'm not too concerned.

No need to guess. The RFC has a mechanism for determining the encoding. It also
specifies a way to indicate the encoding. Which is totally not redundant, in any
way. Thankfully, it doesn't tell us how to handle conflicting information in
this respect, because of course, that would be too easy. Software is no fun when
the specifications make things obvious.

> > 2. How to treat a JSON request with a specified Content-Encoding.
>
> If the encoding is understood, transcode to a UTF-8 representation.

If it is explicit, and it is wrong, vomit in their face.

> > What encodings would be supported?
>
> Patches welcome. UTF-8 currently kinda sort supported.

UTF-8, UTF-16, and UTF-36

> > What would CouchDB do for an unsupported encoding?
>
> Tell the client that we don't support their weirdo character encoding
> and that patches are welcome at the CouchDB JIRA page that no one
> likes visiting cause Java is the devil. Maybe we don't mention that
> last bit though?

Vomit.

> > What would occur if the entity was not consistent with the encoding?
>
> If a client goes out of their way to specify a Content-Encoding and
> they send shit that doesn't comply then we should throw a huge pie at
> them and drop the connection. I'm thinking of a Nelson "Ha, ha!" and
> pointing of many fingers.

I would prefer vomiting, but any kind of humiliation works for me.

> > 3. What should CouchDB send when there is no "Accept-Charset" in the
> > request.
>
> UTF-8. Cause its yummy.

Yum yum for my tum.

> > 4. What should CouchDB send where there is an "Accept-Charset" in the
> > request.  Particularly if the request does not contain a UTF.
> >
>
> If we undersand it, transcode UTF-8 to the requested charset.
> Otherwise, say "Can't do it!".

Vomit.

Best,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
Oh, how do I hate character encoding issues.

Before you get too roped into this, if you can answer this question in
the negative, do so quickly:

Can UTF-8 represent all possible unicode encodings?

I'm gonna assume yes for the rest of this post.

> What Curt said...

I think we're in a bit of a weird spot here cause we're playing with
the head butt of two different RFC's. The HTTP transport RFC that
deals with Content-Encoding and charset awesomeness and the JSON RFC
that is so full of ambiguity I'd like to kick it. On the plus side,
there's so much ambiguity here that we can basically do whatever we
want and no one can accuse us of being wrong.

That said, I think we should isolate concerns. Unless someone want's
to write a JSON parser that understands multiple character encodings
and doesn't suck ass performance wise, we should probably just assume
the JSON parser is UTF-8 only.

Before anyone goes hollering about that, we still have the HTTP layer
to play with in terms of accepting content encoding. And nothing in
the HTTP layer says we have to accept UC-4 or NR-17 or whatever. So
while we're more than welcome to reject any request bodies way before
they hit the JSON serializer, Noah would probably cut my throat for
suggesting we don't play nice. Either way, this big conversation on
character encodings should probably focus on how we move things to
UTF-8 which I officially nominate as the already de-facto CouchDB
character encoding.

> There are a couple of questions that could be addressed:
>
> 1. How to treat a JSON request entity that does not contain a
> Content-Encoding header.  Particularly when the entity is not consistent
> with the expected encoding.

Assume UTF-8. If fail, maybe try guessing. If fail, throw a meatball
at the client saying rejected. We already ignore quite a few headers
and do things "Non-RESTful-ly" so I'm not too concerned.

> 2. How to treat a JSON request with a specified Content-Encoding.

If the encoding is understood, transcode to a UTF-8 representation.

> What encodings would be supported?

Patches welcome. UTF-8 currently kinda sort supported.

> What would CouchDB do for an unsupported encoding?

Tell the client that we don't support their weirdo character encoding
and that patches are welcome at the CouchDB JIRA page that no one
likes visiting cause Java is the devil. Maybe we don't mention that
last bit though?

> What would occur if the entity was not consistent with the encoding?

If a client goes out of their way to specify a Content-Encoding and
they send shit that doesn't comply then we should throw a huge pie at
them and drop the connection. I'm thinking of a Nelson "Ha, ha!" and
pointing of many fingers.

> 3. What should CouchDB send when there is no "Accept-Charset" in the
> request.

UTF-8. Cause its yummy.

> 4. What should CouchDB send where there is an "Accept-Charset" in the
> request.  Particularly if the request does not contain a UTF.
>

If we undersand it, transcode UTF-8 to the requested charset.
Otherwise, say "Can't do it!".

HTH,
Paul Davis

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 01:42:06AM -0400, Paul Davis wrote:
> Thank you Master Yoda for imparting some of your uni-knowledge.

Word up, housey.

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
> Don't get technical on me, motherfuton.
>
> If you want a word for this, you would say they are transcodings of each other.

*LOSSLESS* transcodings

>> But you started it when saying that something that
>> represents a subset of unicode is a valid encoding. Which its not.
>> lrn2nyquistfrequency fool!
>
> US-ASCII encoded an extremely limited subset of Unicode.
>
> US-ASCII is a perfectly valid encoding of Unicode.
>
> Learn to Unicode your face, up in that.

Thank you Master Yoda for imparting some of your uni-knowledge.

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 01:33:14AM -0400, Paul Davis wrote:
> Well technically... If their both valid encodings of unicode, then
> they're both valid encodings of each other. Just because you're
> concerned about how the represent unicode doesn't mean we can't define
> that ISO-8859-1's encoding table isn't encoding utf-8 representations
> of code points. Granted this gets into up is down and left is right
> territory.

Don't get technical on me, motherfuton.

If you want a word for this, you would say they are transcodings of each other.

> But you started it when saying that something that
> represents a subset of unicode is a valid encoding. Which its not.
> lrn2nyquistfrequency fool!

US-ASCII encoded an extremely limited subset of Unicode.

US-ASCII is a perfectly valid encoding of Unicode.

Learn to Unicode your face, up in that.

Kindest regards,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
IRC helped on this.

> What?! Heh.
>
> ISO-8859-1 is an encoding of (a small ass subset of which means not really at all) Unicode.
>
> UTF-8 is an encoding of (ALL) Unicode.

> Neither are valid versions of each other.

Well technically... If their both valid encodings of unicode, then
they're both valid encodings of each other. Just because you're
concerned about how the represent unicode doesn't mean we can't define
that ISO-8859-1's encoding table isn't encoding utf-8 representations
of code points. Granted this gets into up is down and left is right
territory. But you started it when saying that something that
represents a subset of unicode is a valid encoding. Which its not.
lrn2nyquistfrequency fool!

> Neither are content types, or media types, or MIME types.

Analogy fail.

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sun, Aug 30, 2009 at 12:48:36AM -0400, Paul Davis wrote:
> Forgive my ignorance, but I thought ISO-8859-1 was valid UTF-8. Or is
> it just in the common no-high-bit-set ASCII that's UTF-8 compatible?
> Either way, ISO-8859-1 is an encoding, not a content-type. Isn't
> saying it's invalid JSON like saying UC-4 is invalid JPEG? XD

What?! Heh.

ISO-8859-1 is an encoding of Unicode.

UTF-8 is an encoding of Unicode.

Neither are valid versions of each other.

Neither are content types, or media types, or MIME types.

Basically, LOL.

Best,

-- 
Noah Slater, http://tumbolia.org/nslater

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Paul Davis <pa...@gmail.com>.
>  * The editor of RFC 4627 was high.

Just letting this one hang out for awhile now.

>> It still uses the troublesome meme "character encoding ... Unicode",
>> however it seems to be a stretch to read that and think that Shift-JIS,
>> ISO-8559-8, MacLatin, EBCDIC, etc are also fine and dandy.
>
> The RFC demonstrates conclusively that the only allowable encodings are:
>
>  UTF-8, UTF-16, or UTF-32

I missed this. They say fairly explicitly "JSON text SHALL be encoded
in Unicode." To me that says that if I create my own PJD-15.3 Unicode
encoding, a compliant JSON parser must support it. Obviously not gonna
happen, so we should ignore the JSON spec, and rely on our HTTP
prowess to make sure we only feed UTF-8 to the JSON parser.

>   Since the first two characters of a JSON text will always be ASCII
>   characters [RFC0020], it is possible to determine whether an octet
>   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
>   at the pattern of nulls in the first four octets.

Am I the only that has contemplated whether this means the
byte-representation of the JSON object, vs every JSON string in the
object?

> ISO-8859-1 JSON is invalid JSON.

Forgive my ignorance, but I thought ISO-8859-1 was valid UTF-8. Or is
it just in the common no-high-bit-set ASCII that's UTF-8 compatible?
Either way, ISO-8859-1 is an encoding, not a content-type. Isn't
saying it's invalid JSON like saying UC-4 is invalid JPEG? XD

Re: Character encodings and JSON RFC (spun off from COUCHDB-345)

Posted by Noah Slater <ns...@apache.org>.
On Sat, Aug 29, 2009 at 10:51:51PM -0500, Curt Arnold wrote:
> I agree that is unfortunately worded.  I checked the IETF RFC errata
> page (http://www.rfc-editor.org/errata_search.php?rfc=4627) and did not
> find a clarification on this issue.  I basically interpreted "in
> Unicode" as "in a Unicode Transformation Format" and more specifically
> as in a UTF recommended by the Unicode consortium and in widespread use.

  "A string is a sequence of zero or more Unicode characters"

                                           - http://www.ietf.org/rfc/rfc4627.txt

Only possible meaning is any encoding of any Unicode code point.

  "All Unicode characters may be placed within the quotation marks[...]"

                                                                          - ibid

Ditto.

  "JSON text SHALL be encoded in Unicode. The default encoding is UTF-8."

                                                                          - ibid

"encoded in Unicode" is nonsensical, so the only possible parsing is:

  "JSON text SHALL be encoded from Unicode. The default encoding is UTF-8."

                                                                          - ibid

Which means that any encoding is good.

Those are the only mentions of Unicode in the entire specification.

If we search for "encoding" we get:

  "[...] the character is represented as a twelve-character sequence,
   encoding the UTF-16 surrogate pair"

                                                                          - ibid

  "JSON may be represented using UTF-8, UTF-16, or UTF-32. When JSON is written
  in UTF-8, JSON is 8bit compatible. When JSON is written in UTF-16 or UTF-32,
  the binary content-transfer-encoding must be used."

                                                                          - ibid

>From this we can conclude:

  * JSON can be encoded in UTF-8, UTF-16, or UTF-32.

  * The editor of RFC 4627 was high.

> There is this quote from http://www.json.org/fatfree.html:
>>
>> The character encoding of JSON text is always Unicode. UTF-8 is the
>> only encoding that makes sense on the wire, but UTF-16 and UTF-32 are
>> also permitted.
>
> It still uses the troublesome meme "character encoding ... Unicode",
> however it seems to be a stretch to read that and think that Shift-JIS,
> ISO-8559-8, MacLatin, EBCDIC, etc are also fine and dandy.

The RFC demonstrates conclusively that the only allowable encodings are:

  UTF-8, UTF-16, or UTF-32

> Don Box also seems to have a similar interpretation
> (http://www.pluralsight.com/community/blogs/dbox/archive/2007/01/03/45560.aspx
> ):
>>
>> 2. Like Joe Gregorio states in the comments on Tim's post, I also
>> prefer JSON's simplification of only allowing UTF-*-based encoding.
>
> Tim Bray in http://www.tbray.org/ongoing/When/200x/2006/12/21/JSON had
> an interesting comment in:
>>
>> I look at the Ruby JSON library, for example, and I see all these
>> character encoding conversion routines; blecch.
>> Use JSON · Seems easy to me; if you want to serialize a data structure
>> that’s not too text-heavy and all you want is for the receiver to get
>> the same data structure with minimal effort, and you trust the other
>> end to get the i18n right, JSON is hunky-dory.
>
> This hints things in the field aren't all pristine UTF encodings
> however.
>
> Probably best to ping the RFC editor to see if there is a clarification.

UTF-8, UTF-16, or UTF-32.

>> In this case, what the JSON RFC should say is that JSON should be
>> encoded from Unicode, which means that the encoding could be anything
>> from ISO-8859-1 to Shift JIS, which means that we cannot
>> "unambiguously determine the encoding from the content." Even if we
>> decided to only allow UTF-8, UTF-16, or UTF-32, we could only
>> "unambiguously determine the encoding" if the request body included
>> the BOM, which is entirely optional. So again, without the Content-
>> Encoding information, we are forced to use a heuristic. Heuristics
>> already exist, and where they are not already available in Erlang, I
>> rather suspect that they can be ported with relative ease.
>
> I haven't worked through all the sequences, but since we know that the
> first characters of a JSON string is either " " or "{", that should be
> enough to unambiguously determine whether the only possible encoding of
> the set UTF-8, UTF-16BE, UTF-16LE, or UTF-32.

   Since the first two characters of a JSON text will always be ASCII
   characters [RFC0020], it is possible to determine whether an octet
   stream is UTF-8, UTF-16 (BE or LE), or UTF-32 (BE or LE) by looking
   at the pattern of nulls in the first four octets.

           00 00 00 xx  UTF-32BE
           00 xx 00 xx  UTF-16BE
           xx 00 00 00  UTF-32LE
           xx 00 xx 00  UTF-16LE
           xx xx xx xx  UTF-8

                                                                          - ibid

>> If we can access the Content-Encoding, we should absolutely use it,
>> and absolutely reject as garbage any request that could not be decoded
>> with the explicit encoding. Any patch that will-fully ignored this
>> information only to fall back onto a heuristic would get my emphatic
>> veto. I am however satisfied with requiring UTF-8 in the short term,
>> and adding Content-Encoding awareness at some later point.
>
> The RFC says "A JSON parser MAY accept non-JSON forms or extensions."
> So unlike an XML processor that is prohibited from assuming ISO-8859-1
> if it encounters a invalid UTF-8 sequence, a JSON parser could
> transparently assume ISO-8859-1 after encountering a bad UTF-8 sequence.
> Whether that would be a good thing  is debatable.

ISO-8859-1 JSON is invalid JSON.

> There are a couple of questions that could be addressed:
>
> 1. How to treat a JSON request entity that does not contain a Content-
> Encoding header.  Particularly when the entity is not consistent with
> the expected encoding.

Considering we can reliably determine the encoding, we can:

  * Ignore the Content-Encoding header completely.

  * Reject any request where the Content-Encoding header is wrong.

The jerk in me wants to opt for the second option, but there is always Postel's.

> 2. How to treat a JSON request with a specified Content-Encoding.  What
> encodings would be supported?  What would CouchDB do for an unsupported
> encoding?  What would occur if the entity was not consistent with the
> encoding?

Ignore it or barf. See above dilemma of jerk vs. hippie approach.

UTF-8, UTF-16, or UTF-32.

Barf.

Ignore it or barf. As above.

> 3. What should CouchDB send when there is no "Accept-Charset" in the
> request.

UTF-8.

> 4. What should CouchDB send where there is an "Accept-Charset" in the
> request.  Particularly if the request does not contain a UTF.

406 Not acceptable.

> I think the current answers are:
>
> 1. Entity is interpreted as UTF-8.  Currently if the encoding is
> inconsistent, it is still committed to the database and bad things
> happen later.  If a fix for COUCHDB-345 is committed, then CouchDB would
> reject the request with a 400.

+1

> 2. Same as 1, Content-Encoding is not considered.

Undecided.

> 3. CouchDB always sends UTF-8.

+1

> 4. Same as 3, Accept-Charset is not considered.

-1

Come on, you gotta give me this. It's fun to send back 406! Stupid clients.

> It is not a pressing issue for me and since COUCHDB-345 languished for
> such a long time, I'm not thinking that many people are trying to push
> other encodings into the database with the exceptions of people pushing
> ISO-8859-1 up but not getting burned since their content hasn't yet
> contained non ASCII characters.

I apologise for not reading to (totally bonkers) RFC properly until now.

Did I mention it's totally bonkers?

Best,

-- 
Noah Slater, http://tumbolia.org/nslater