You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Chris Anderson <jc...@apache.org> on 2009/06/22 08:21:00 UTC
Unicode normalization (was Re: The 1.0 Thread)
On Sun, Jun 21, 2009 at 4:40 PM, Antony Blakey<an...@gmail.com> wrote:
>
> On 22/06/2009, at 7:26 AM, Paul Davis wrote:
>
>> Also +lots on deterministic revisions. As a side note, we've been
>> worrying a bit about how to calculate the revision id's in the face of
>> JSON differences in clients. I make a motion that we just stop caring
>> and define how we calculate the signature. Ie, instead of calling it
>> canonical JSON just call it, "The CouchDB deterministic revision
>> algorithm" or some such. Then we can feel free to punt on any of the
>> awesome issues we'd run into with a full canonical JSON standard.
>
> I haven't seen the recent discussions about canonicalisation, but IMO a
> minimum requirement is that the infoset <-> serialisation mapping must be
> 1:1, which requires completeness and prescription. Doing unicode
> normalisation (NFC probably) is IMO also an absolute requirement - it's
> virtually impossible to construct documents by hand with repeatable results
> without it.
>
My gut reaction is that normalizing strings using NFC [1] is not
appropriate for a database. Here's why we should treat strings as
binary and not worry about unicode normalization at all:
First of all, I'm certain we can't require that all input already be
NFC normalized. The real-life failure condition would be: "your
language / operating system is not supported by CouchDB." A normal
user is not going to understand the first bit of the fact that the
underlying binary representation of their text could be subtly
different in a way that would be invisible to them. And if they did
understand that, they'd be hard pressed to change it. So rejecting
non-normalized strings is unacceptable.
Secondly, we're a database, so I find highly suspicious the notion
that we should auto-normalize user input on-the-quiet. Maybe
normalization is not lossy, but one particular use-case (however slim)
that we can't support if we auto-normalize is a document which lists
variations on the same string, to illustrate how non-normalized forms
look the same but have different binary representations. A database
which can't store that seems flawed, to me.
So we can't require normalized input and we can't auto-normalize.
Where does this leave us?
Under the current (raw binary) string handling, two variations on a
document which would, when NFC normalized, be binary identical, could
have different deterministic revs. Since 99+% of content is already
normalized, we're looking at a very small set of cases where we'd have
distinct revs for documents that have similar (or identical, depending
on your pov) content. The fact that there are rare pairs of documents
out there which one could argue are the same, but which have different
revs, strikes me as ever so slightly non-optimal, but not really a big
deal.
I think the potential optimization can be nicely accounted for by a
simple recommendation:
* If you are doing independent updates (from distinct client software)
of strings in a document, and relying on deterministic revs to avoid
conflict-on-replication, you should NFC normalize your content.
The antecedents in that clause show how the case where normalization
matters for deterministic revs is even more rare than the existence of
non-normalized unicode. A secondary recommendation for people relying
on deterministic revs to avoid conflicts on multi-node-updates, would
be:
"don't mutilate strings you didn't edit" so as long as client software
doesn't go jiggling forms to other random look-alike codepoints
without asking, any potential trouble is confined to fields actually
effected by an update.
The common use case for these revs is not lots of distinct client
softwares all doing identical updates by hand and then pushing them to
different eventually-replicating CouchDB cluster members. (Which is
the use case where any of the above discussion is relevant.)
The paradigm use case of deterministic revs is a single piece of
software, running on a single box, creating a document and saving it
to multiple cluster members using the same rev. Treating strings as
binary completely and totally serves this use case.
Chris
[1] http://www.macchiato.com/unicode/nfc-faq
> Unicode normalisation is an issue for clients because it requires they have
> access to a Unicode NFC function.
>
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> It is as useless to argue with those who have renounced the use of reason as
> to administer medication to the dead.
> -- Thomas Jefferson
>
>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
Couple things over thinking this over for a night.
Firstly, I was about to write an email almost exactly like Chris' last
night, but during the time of drafting it I started looking into
unicode normalization and what effects it might have. As it turns out,
we're already messing around with strings in such a way as to be
confusing. For instance, unicode escaped latin characters are mutated
when going through the JSON decoder. As in, "\u0043" -> "C".
Secondly, as Noah points out, we shouldn't be using the derived type
for storage, only for the revision calculation, and as Anthony points
out, we could always implement an endpoint to make the normalization
algorithm available to clients. And lest we forget, there are also
major headaches awaiting us with float normalization.
Yay fun stuff!
Paul
On Mon, Jun 22, 2009 at 11:19 AM, Antony Blakey<an...@gmail.com> wrote:
>
> On 23/06/2009, at 12:06 AM, Noah Slater wrote:
>
>> On Sun, Jun 21, 2009 at 11:21:00PM -0700, Chris Anderson wrote:
>>>
>>> My gut reaction is that normalizing strings using NFC [1] is not
>>> appropriate
>>> for a database. Here's why we should treat strings as binary and not
>>> worry
>>> about unicode normalization at all:
>>
>> [...]
>>>
>>> First of all, I'm certain we can't require that all input already be NFC
>>> normalized.
>>
>> [...]
>>>
>>> Secondly, we're a database, so I find highly suspicious the notion that
>>> we
>>> should auto-normalize user input on-the-quiet.
>>
>> [...]
>>>
>>> So we can't require normalized input and we can't auto-normalize.
>>
>> CouchDB would create a canonicalised copy of the document while creating
>> the
>> document hash. There is no reason why CouchDB, or the clients, should
>> worry
>> about canonicalising the actual documents.
>>
>>> Where does this leave us?
>>
>> Canonicalisation is a temporary step, so there are no problems.
>
> +1 to those two points.
>
>>>> Unicode normalisation is an issue for clients because it requires they
>>>> have
>>>> access to a Unicode NFC function.
>>
>> Why would clients need to worry about this? CouchDB is creating the
>> hashes.
>
> At the moment, sure, but I was anticipating cases where this the canonical
> form, or a hash thereof would then creep into other contexts i.e. once you
> have the facility, who knows what you might want to do. OTOH, this could be
> dealt with via a canonicalisation service e.g. POST json payload(s), get
> back hashes of the canonical form(s) (or the forms themselves), which means
> that systems without access to unicode normalisation can still function with
> future facilities.
>
> Antony Blakey
> --------------------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> The greatest challenge to any thinker is stating the problem in a way that
> will allow a solution
> -- Bertrand Russell
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Antony Blakey <an...@gmail.com>.
On 23/06/2009, at 12:06 AM, Noah Slater wrote:
> On Sun, Jun 21, 2009 at 11:21:00PM -0700, Chris Anderson wrote:
>> My gut reaction is that normalizing strings using NFC [1] is not
>> appropriate
>> for a database. Here's why we should treat strings as binary and
>> not worry
>> about unicode normalization at all:
> [...]
>> First of all, I'm certain we can't require that all input already
>> be NFC
>> normalized.
> [...]
>> Secondly, we're a database, so I find highly suspicious the notion
>> that we
>> should auto-normalize user input on-the-quiet.
> [...]
>> So we can't require normalized input and we can't auto-normalize.
>
> CouchDB would create a canonicalised copy of the document while
> creating the
> document hash. There is no reason why CouchDB, or the clients,
> should worry
> about canonicalising the actual documents.
>
>> Where does this leave us?
>
> Canonicalisation is a temporary step, so there are no problems.
+1 to those two points.
>>> Unicode normalisation is an issue for clients because it requires
>>> they have
>>> access to a Unicode NFC function.
>
> Why would clients need to worry about this? CouchDB is creating the
> hashes.
At the moment, sure, but I was anticipating cases where this the
canonical form, or a hash thereof would then creep into other contexts
i.e. once you have the facility, who knows what you might want to do.
OTOH, this could be dealt with via a canonicalisation service e.g.
POST json payload(s), get back hashes of the canonical form(s) (or the
forms themselves), which means that systems without access to unicode
normalisation can still function with future facilities.
Antony Blakey
--------------------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
The greatest challenge to any thinker is stating the problem in a way
that will allow a solution
-- Bertrand Russell
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Antony Blakey <an...@gmail.com>.
On 24/06/2009, at 1:57 AM, Paul Davis wrote:
> Are there byte order semantics for UTF-8?
No, UTF-8 is independent of byte ordering because it's a byte stream.
> Or other cases where sorting
> by UTF-8 binary representation is going to cause issues? Remember that
> the end goal is to create deterministic serializations for hashing.
Sorting over the UTF-8 bytes is fine for this.
> Sorting by code point doesn't seem like it'd get us anything other
> than added complexity.
Agreed, because you would have to deal with surrogates in UTF-16.
> Patches welcome.
Of course. Let me qualify by saying I have no time to do this, I'm
merely taking part in a discussion.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
When I hear somebody sigh, 'Life is hard,' I am always tempted to ask,
'Compared to what?'
-- Sydney Harris
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Antony Blakey <an...@gmail.com>.
On 24/06/2009, at 1:57 AM, Paul Davis wrote:
> Are there byte order semantics for UTF-8?
No, UTF-8 is independent of byte ordering because it's a byte stream.
> Or other cases where sorting
> by UTF-8 binary representation is going to cause issues? Remember that
> the end goal is to create deterministic serializations for hashing.
Sorting over the UTF-8 bytes is fine for this.
> Sorting by code point doesn't seem like it'd get us anything other
> than added complexity.
Agreed, because you would have to deal with surrogates in UTF-16.
> Patches welcome.
Of course. Let me qualify by saying I have no time to do this, I'm
merely taking part in a discussion.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
When I hear somebody sigh, 'Life is hard,' I am always tempted to ask,
'Compared to what?'
-- Sydney Harris
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 23, 2009 at 11:15 AM, Antony Blakey<an...@gmail.com> wrote:
>
> On 23/06/2009, at 11:43 PM, Paul Davis wrote:
>
>> Interesting point. I take this as a pretty clear reason for discarding
>> UCA for member ordering. Normalization isn't affected by locale right?
>> I haven't seen anything to suggest as such so I assume not.
>
> IIUC each normalization form is strictly functional.
>
> IMO, given that ICU provides normalization functions, CouchDB should use
> them in this case, exposing the canonicalisation transformation, and a
> shortcut producing the hash, as a client-accessible endpoint. I say this
> from a 'why not do it right?' perspective.
>
Regardless of whether normalization is included in the deterministic
revision algorithm I think that normalization end points, or perhaps
query string parameters for performing normalization on incoming data
would be a good thing.
I reject the assertion that this is a question of right or wrong. I
see valid arguments for and against including normalization as part of
the deterministic revision algorithm. Considering a query parameter to
process incoming data conditionally I'm leaning towards not including
it, but I'm always open to being convinced otherwise.
> Member ordering could be binary, over either the code points (e.g. 32 bits)
> or the bytes of the UTF-8 representation. Given the ease of creating a UTF-8
> iterator that is probably best. UTF-16 is the most common native encoding,
> but you don't want to do a byte-level collation over a UTF-16/32 encoding
> because the result is dependent on byte ordering.
>
Are there byte order semantics for UTF-8? Or other cases where sorting
by UTF-8 binary representation is going to cause issues? Remember that
the end goal is to create deterministic serializations for hashing.
Sorting by code point doesn't seem like it'd get us anything other
than added complexity.
> The problem with this is that the canonical form might look bizarre for a
> non-ASCII document, but a canonical collation is by definition always going
> to look wrong to someone. For the current use, as an intermediate form
> destined only for hashing, this doesn't matter anyway.
>
Yep.
> Having said that, IMO it would be a good i18n feature to be able set the
> locale of a database, maybe even at the granularity of a view, defaulting to
> the database's locale. The key ordering should respect that locale. An
> option to normalize keys would also be a good idea. The reason for setting a
> locale at the view level is that it might be useful to create multiple views
> with different locales, to present different localized result orderings to
> end users. One immediate issue is that the local would have to be injected
> into view servers to prevent possible weirdness.
>
> I think it's easier and better to do these kind of things on the server
> because you know you have the facilities to do it there (e.g. ICU), whereas
> making it a client issue impedes use of the data by different clients.
>
Patches welcome.
Paul Davis
> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> On the other side, you have the customer and/or user, and they tend to do
> what we call "automating the pain." They say, "What is it we're doing now?
> How would that look if we automated it?" Whereas, what the design process
> should properly be is one of saying, "What are the goals we're trying to
> accomplish and how can we get rid of all this task crap?"
> -- Alan Cooper
>
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Wed, Jun 24, 2009 at 12:45:58AM +0930, Antony Blakey wrote:
> Having said that, IMO it would be a good i18n feature to be able set the
> locale of a database, maybe even at the granularity of a view, defaulting to
> the database's locale. The key ordering should respect that locale. An option
> to normalize keys would also be a good idea. The reason for setting a locale
> at the view level is that it might be useful to create multiple views with
> different locales, to present different localized result orderings to end
> users. One immediate issue is that the local would have to be injected into
> view servers to prevent possible weirdness.
+1 on all of this.
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Antony Blakey <an...@gmail.com>.
On 23/06/2009, at 11:43 PM, Paul Davis wrote:
> Interesting point. I take this as a pretty clear reason for discarding
> UCA for member ordering. Normalization isn't affected by locale right?
> I haven't seen anything to suggest as such so I assume not.
IIUC each normalization form is strictly functional.
IMO, given that ICU provides normalization functions, CouchDB should
use them in this case, exposing the canonicalisation transformation,
and a shortcut producing the hash, as a client-accessible endpoint. I
say this from a 'why not do it right?' perspective.
Member ordering could be binary, over either the code points (e.g. 32
bits) or the bytes of the UTF-8 representation. Given the ease of
creating a UTF-8 iterator that is probably best. UTF-16 is the most
common native encoding, but you don't want to do a byte-level
collation over a UTF-16/32 encoding because the result is dependent on
byte ordering.
The problem with this is that the canonical form might look bizarre
for a non-ASCII document, but a canonical collation is by definition
always going to look wrong to someone. For the current use, as an
intermediate form destined only for hashing, this doesn't matter anyway.
Having said that, IMO it would be a good i18n feature to be able set
the locale of a database, maybe even at the granularity of a view,
defaulting to the database's locale. The key ordering should respect
that locale. An option to normalize keys would also be a good idea.
The reason for setting a locale at the view level is that it might be
useful to create multiple views with different locales, to present
different localized result orderings to end users. One immediate issue
is that the local would have to be injected into view servers to
prevent possible weirdness.
I think it's easier and better to do these kind of things on the
server because you know you have the facilities to do it there (e.g.
ICU), whereas making it a client issue impedes use of the data by
different clients.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
On the other side, you have the customer and/or user, and they tend to
do what we call "automating the pain." They say, "What is it we're
doing now? How would that look if we automated it?" Whereas, what the
design process should properly be is one of saying, "What are the
goals we're trying to accomplish and how can we get rid of all this
task crap?"
-- Alan Cooper
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 23, 2009 at 5:05 AM, Antony Blakey<an...@gmail.com> wrote:
>
> On 23/06/2009, at 11:22 AM, Noah Slater wrote:
>
>> Could Unicode normalisation affect the sorting of keys with option two?
>
> On a related issue: the locale certainly could e.g. it changes the collation
> order of accented characters.
>
Interesting point. I take this as a pretty clear reason for discarding
UCA for member ordering. Normalization isn't affected by locale right?
I haven't seen anything to suggest as such so I assume not.
> Antony Blakey
> -------------
> CTO, Linkuistics Pty Ltd
> Ph: 0438 840 787
>
> On the other side, you have the customer and/or user, and they tend to do
> what we call "automating the pain." They say, "What is it we're doing now?
> How would that look if we automated it?" Whereas, what the design process
> should properly be is one of saying, "What are the goals we're trying to
> accomplish and how can we get rid of all this task crap?"
> -- Alan Cooper
>
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Antony Blakey <an...@gmail.com>.
On 23/06/2009, at 11:22 AM, Noah Slater wrote:
> Could Unicode normalisation affect the sorting of keys with option
> two?
On a related issue: the locale certainly could e.g. it changes the
collation order of accented characters.
Antony Blakey
-------------
CTO, Linkuistics Pty Ltd
Ph: 0438 840 787
On the other side, you have the customer and/or user, and they tend to
do what we call "automating the pain." They say, "What is it we're
doing now? How would that look if we automated it?" Whereas, what the
design process should properly be is one of saying, "What are the
goals we're trying to accomplish and how can we get rid of all this
task crap?"
-- Alan Cooper
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jun 22, 2009 at 9:52 PM, Noah Slater<ns...@apache.org> wrote:
> On Mon, Jun 22, 2009 at 04:22:10PM -0400, Paul Davis wrote:
>> On Mon, Jun 22, 2009 at 3:32 PM, Noah Slater<ns...@apache.org> wrote:
>> >
>> > * calculate the document hash from a canonical binary serialisation
>> >
>> > * calculate the document hash from the binary serialisation
> [...]
>>
>> * calculate the document hash from the deterministic binary serialization
>
> Okay, so we have three options:
>
> * hash the document, binary identical to what the client sent
>
> * hash the document, with sorted keys, but binary identical key/value data
>
> * hash the document, with sorted keys, and Unicode normalised key/value data
>
> This raise the obvious question:
>
> Could Unicode normalisation affect the sorting of keys with option two?
>
Noper.
UCA Step 1: Convert input strings to Normalized Form D
http://unicode.org/reports/tr10/#Main_Algorithm
>> Obviously, regardless of what we choose to implement, safe-writes are kosher
>> because all the server is the only one ever doing the actual calculation. The
>> unicode normalization issue crops up if two different clients write the same
>> edit to multiple hosts. If the clients don't use the same normalization scheme
>> then we still introduce the spurious conflicts (that would currently be
>> introduced no matter what) on replication.
>
> Nope, this is exactly what Unicode normalisation is designed to solve. If two
> different clients write ESSENTIALLY the same edit to two different nodes using
> two different methods of combining characters, then Unicode normalisation will
> see these as the same edit. Without normalisation, these two will have a
> different document hash, and will conflict.
>
Slow down a bit before you discard out of hand. That is precisely what
I'm saying: we might not prevent conflicts in every case that we could
if we included normalization. Although, the fun part of this whole
exercise is that we can all have our own awesomely independent
interpretation of what "essentially the same edit" means. The explicit
question we are considering is this:
Is the code point U+00C7 equal to the code point sequence U+0043 U+0327 ?
The two answers I see are:
Yes - Unicode defines U+00C7 as a canonical equivalent of U+0043 U+0327
No - The UTF-8 byte representation of the two sequences are different.
Reasons for implementing according to Yes:
* We avoid the maximum number of spurious conflicts in the
safe-writes example without requiring all clients to implement unicode
normalization.
Reasons for implementing according to No:
* Anyone implementing the deterministic revision algorithm MUST do
unicode normalization.
* Clients are still free to send normalized unicode thus receiving
the previously described benefits.
* It may be possible that there's a use case in which it is
desirable for the different code point combinations to result in a
conflict.
* Performance (Ie, the same reason _all_docs no longer uses UCA)
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
As an aside, as I read the Unicode FAQ at [1] we are not required to
recognize the canonical equivalence of code point sequences.
[1] http://unicode.org/faq/char_combmark.html#8
HTH,
Paul Davis
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Mon, Jun 22, 2009 at 04:22:10PM -0400, Paul Davis wrote:
> On Mon, Jun 22, 2009 at 3:32 PM, Noah Slater<ns...@apache.org> wrote:
> >
> > * calculate the document hash from a canonical binary serialisation
> >
> > * calculate the document hash from the binary serialisation
[...]
>
> * calculate the document hash from the deterministic binary serialization
Okay, so we have three options:
* hash the document, binary identical to what the client sent
* hash the document, with sorted keys, but binary identical key/value data
* hash the document, with sorted keys, and Unicode normalised key/value data
This raise the obvious question:
Could Unicode normalisation affect the sorting of keys with option two?
Anyone know much about collation algorithms?
> Obviously, regardless of what we choose to implement, safe-writes are kosher
> because all the server is the only one ever doing the actual calculation. The
> unicode normalization issue crops up if two different clients write the same
> edit to multiple hosts. If the clients don't use the same normalization scheme
> then we still introduce the spurious conflicts (that would currently be
> introduced no matter what) on replication.
Nope, this is exactly what Unicode normalisation is designed to solve. If two
different clients write ESSENTIALLY the same edit to two different nodes using
two different methods of combining characters, then Unicode normalisation will
see these as the same edit. Without normalisation, these two will have a
different document hash, and will conflict.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Chris Anderson <jc...@apache.org>.
On Mon, Jun 22, 2009 at 1:22 PM, Paul Davis<pa...@gmail.com> wrote:
> On Mon, Jun 22, 2009 at 3:32 PM, Noah Slater<ns...@apache.org> wrote:
>> On Mon, Jun 22, 2009 at 03:15:24PM -0400, Paul Davis wrote:
>
> Exactly, though I would add a third choice that is
>
> * calculate the document hash from the deterministic binary serialization
I think this is the 90% solution. Unicode normalization may be the
other 10%. I just don't want to see the last 10% block the first 90.
>
> Which would include requirements like serializing document members
> with some defined ordering.
>
> On a side note, I've also contemplated just hashing the incoming
> binary representation as the new revision. Though that comes with its
> own set of issues obviously.
>
I don't have anything against unicode normalization, I just don't
think it buys us a *whole* lot. I do think recursive sorting and
deterministic float handling are pretty crucial, as even the same Ruby
client will order the keys differently on subsequent PUTs. Just
hashing the PUT body would not be sufficient, I think. It'd be more
like a 50% solution.
Here's some code a wrote a while ago that handles floats and key sorting in JS:
http://github.com/jchris/canonical-json/blob/83751a8b650c60a5fcf3ed4ad5337e3dd172b521/test.html
The client should not send the string used for hashing as the document
itself, the hash would be made from a string derived from the
document, which would be lossy on floats. In Couch we'd want to do
this in Erlang, and not store any effects of the function except the
hash.
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jun 22, 2009 at 3:32 PM, Noah Slater<ns...@apache.org> wrote:
> On Mon, Jun 22, 2009 at 03:15:24PM -0400, Paul Davis wrote:
>> I think he means optimization in so much as that the deterministic
>> revision algorithm still works regardless.
>
> For what definition of "works" though?
>
> Should two versions of the same document, one using combining code points and
> the other using single code points, generate the same hash?
I dunno. I would lean towards the revision system not caring about
unicode and not caring about canonical JSON. If there's a desire for
it we could provide an end point that does normalization in the same
style that we provide a uuids endpoint for clients with no good source
of entropy.
> If the answer is no,
> then why don't we drop the pretence of working with a canonical version of the
> document and just SHA1 the binary serialisation?
>
Right. Quoting myself from the original thread:
Ie, instead of calling it canonical JSON just call it, "The CouchDB
deterministic revision
algorithm" or some such.
>> If clients want to avoid spurious conflicts, then they should send normalized
>> unicode to avoid the issue.
>
> You're arguing for using a blind a binary hash like SHA1, and putting the onus
> on clients to perform canonicalisation. That's fine, as long as you realise that
> this isn't what was originally being proposed.
>
>> In other words, if we just write the algorithm to not care about normalization
>> it'll solve lots of cases for free and the cases that aren't solved can be
>> solved if the client so desires.
>
> I'm not so sure what this "solves" other than implementation effort for us.
>
Perhaps solve is the wrong verb. I'm saying that in my experience,
alot of the use cases of deterministic hashing would work correctly
without fretting about unicode normalization. As Chris points out, we
may have conflicts in cases where normalization would have prevented
them, but as it is, those are already conflicts no matter what. Also,
if a client wants to go the extra mile and provide normalized unicode
then they will even avoid this situation, but not at the expense of
neglecting any client that is unable to provide normalized unicode due
to environment.
> Again, it seems like you're wanting to offer to routes for us:
>
> * calculate the document hash from a canonical binary serialisation
>
> * calculate the document hash from the binary serialisation
>
> Both of these are reasonable choices, depending on our goals. But we need to
> realise that JSON canonicalisation REQUIRES Unicode canonicalisation, and so the
> choice isn't about ignoring Unicode issues, it's about deciding what a canonical
> serialisation hash buys us above and beyond a totally blind one.
>
Exactly, though I would add a third choice that is
* calculate the document hash from the deterministic binary serialization
Which would include requirements like serializing document members
with some defined ordering.
On a side note, I've also contemplated just hashing the incoming
binary representation as the new revision. Though that comes with its
own set of issues obviously.
>> The thing that worries me most about normalization is that we would end up
>> causing more problems by being complete than if we just took the naive route.
>> Requiring that a client have an implementation of normalization byte identical
>> to the one CouchDB uses instead of just being internally consistent seems like
>> it could trip up alot of clients.
>
> Interestingly enough, your suggestion to use a blind binary hash such as SHA1
> pushes the canonicalisation issues onto the client, and would force them to find
> a complete Unicode library. If CouchDB calculated the hash from a canonical
> serialisation internally, we remove this burden from the clients.
>
Time to back up a bit I think.
For me there are basically two main use cases for which we want
deterministic revisioning:
1. A client writes the same edit to multiple couchdb nodes. When the
nodes replicate, no spurious conflict is introduced.
2. A client has knowledge of a document, and want's to be able to
calculate new revisions client side.
For clarity, I'll call the first example the 'safe-writes' example,
and the second will be 'algorithm-implementer'.
Obviously, regardless of what we choose to implement, safe-writes are
kosher because all the server is the only one ever doing the actual
calculation. The unicode normalization issue crops up if two different
clients write the same edit to multiple hosts. If the clients don't
use the same normalization scheme then we still introduce the spurious
conflicts (that would currently be introduced no matter what) on
replication.
For the algorithm-implementer example if we require normalziation,
then all clients wishing to calculate revisions must include unicode
normalization.
What worries me is that the algorithm-implementers would have a
disproportionately more complicated implementation, especially when
the benefit to the safe-writes example could be slim to none assuming
a homogeneous client implementation.
A trade off would be to have an option to do normalization in CouchDB.
Whether it be a flag when writing records or a URL endpoint.
I'm tired of typing.
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Mon, Jun 22, 2009 at 03:15:24PM -0400, Paul Davis wrote:
> I think he means optimization in so much as that the deterministic
> revision algorithm still works regardless.
For what definition of "works" though?
Should two versions of the same document, one using combining code points and
the other using single code points, generate the same hash? If the answer is no,
then why don't we drop the pretence of working with a canonical version of the
document and just SHA1 the binary serialisation?
> If clients want to avoid spurious conflicts, then they should send normalized
> unicode to avoid the issue.
You're arguing for using a blind a binary hash like SHA1, and putting the onus
on clients to perform canonicalisation. That's fine, as long as you realise that
this isn't what was originally being proposed.
> In other words, if we just write the algorithm to not care about normalization
> it'll solve lots of cases for free and the cases that aren't solved can be
> solved if the client so desires.
I'm not so sure what this "solves" other than implementation effort for us.
Again, it seems like you're wanting to offer to routes for us:
* calculate the document hash from a canonical binary serialisation
* calculate the document hash from the binary serialisation
Both of these are reasonable choices, depending on our goals. But we need to
realise that JSON canonicalisation REQUIRES Unicode canonicalisation, and so the
choice isn't about ignoring Unicode issues, it's about deciding what a canonical
serialisation hash buys us above and beyond a totally blind one.
> The thing that worries me most about normalization is that we would end up
> causing more problems by being complete than if we just took the naive route.
> Requiring that a client have an implementation of normalization byte identical
> to the one CouchDB uses instead of just being internally consistent seems like
> it could trip up alot of clients.
Interestingly enough, your suggestion to use a blind binary hash such as SHA1
pushes the canonicalisation issues onto the client, and would force them to find
a complete Unicode library. If CouchDB calculated the hash from a canonical
serialisation internally, we remove this burden from the clients.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Mon, Jun 22, 2009 at 2:46 PM, Noah Slater<ns...@apache.org> wrote:
> On Mon, Jun 22, 2009 at 09:21:44AM -0700, Chris Anderson wrote:
>> My larger point is that normalization is basically an optimization.
>
> Optimisation of what? Unicode normalisation should be considered absolutely
> critical to any canonical form. If we want to use some proprietary algorithm for
> determining a document hash, then fine - but if we are advertising that it is
> calculated from some canonical serialisation, then Unicode normalisation really
> is a base requirement for that.
>
I think he means optimization in so much as that the deterministic
revision algorithm still works regardless. If clients want to avoid
spurious conflicts, then they should send normalized unicode to avoid
the issue. In other words, if we just write the algorithm to not care
about normalization it'll solve lots of cases for free and the cases
that aren't solved can be solved if the client so desires.
>> Occasionally getting the hash wrong (for whatever reason) is just
>> going to result in spurious conflicts, which aren't critical errors,
>> just an annoyance as the application will sometimes have to repair the
>> conflicts. Presumably you'll already be repairing conflicts, so fixing
>> ones that result from spurious conflicts (especially as they are so
>> rare) is not a big cost, and should be easy as the application can
>> just pick a random version of the doc, with no ill effects.
>
> As native English speakers, it's fairly easy for us to assume that most
> documents are comprised of some simple Latin character subset. As soon as you
> start working with languages that make heavy use of combining characters -
> accents, diacritical marks, etc - then this character normalisation becomes a
> major issue in a multi-user environment.
>
> Consider the following byte sequences:
>
> U+006B U+014D U+0061 U+006E
>
> U+006B U+006F U+0304 U+0061 U+006E
>
> Both of these look like "kōan" yet the byte sequence depends on my input method.
>
The question is what are we trying to accomplish. We could sit around
and argue whether we're creating revisions of the sequences of code
points or sequences of characters. I'm more than happy to say
sequences of bytes (and thus code points) because that is applicable
to most clients and does not prevent the algorithm from working on
normalized unicode.
>> I think this makes the NFC stuff a nice-to-have, not a necessity.
>
> I disagree strongly.
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
The thing that worries me most about normalization is that we would
end up causing more problems by being complete than if we just took
the naive route. Requiring that a client have an implementation of
normalization byte identical to the one CouchDB uses instead of just
being internally consistent seems like it could trip up alot of
clients. Granted there's a lot of other corners to get snagged on, so
perhaps for the time being we should just say screw it, implement
something and see what clients have problems. Even adding
normalization as part of the step or not I could go with a coin toss
at this point assuming our ICU library dependency can do it.
Either way, enough hand waving for one day.
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Mon, Jun 22, 2009 at 07:46:40PM +0100, Noah Slater wrote:
> Consider the following byte sequences:
>
> U+006B U+014D U+0061 U+006E
>
> U+006B U+006F U+0304 U+0061 U+006E
>
> Both of these look like "kōan" yet the byte sequence depends on my input method.
Just to be pedantic, these are code points. Heh.
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Mon, Jun 22, 2009 at 09:21:44AM -0700, Chris Anderson wrote:
> My larger point is that normalization is basically an optimization.
Optimisation of what? Unicode normalisation should be considered absolutely
critical to any canonical form. If we want to use some proprietary algorithm for
determining a document hash, then fine - but if we are advertising that it is
calculated from some canonical serialisation, then Unicode normalisation really
is a base requirement for that.
> Occasionally getting the hash wrong (for whatever reason) is just
> going to result in spurious conflicts, which aren't critical errors,
> just an annoyance as the application will sometimes have to repair the
> conflicts. Presumably you'll already be repairing conflicts, so fixing
> ones that result from spurious conflicts (especially as they are so
> rare) is not a big cost, and should be easy as the application can
> just pick a random version of the doc, with no ill effects.
As native English speakers, it's fairly easy for us to assume that most
documents are comprised of some simple Latin character subset. As soon as you
start working with languages that make heavy use of combining characters -
accents, diacritical marks, etc - then this character normalisation becomes a
major issue in a multi-user environment.
Consider the following byte sequences:
U+006B U+014D U+0061 U+006E
U+006B U+006F U+0304 U+0061 U+006E
Both of these look like "kōan" yet the byte sequence depends on my input method.
> I think this makes the NFC stuff a nice-to-have, not a necessity.
I disagree strongly.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Chris Anderson <jc...@apache.org>.
On Mon, Jun 22, 2009 at 7:36 AM, Noah Slater<ns...@apache.org> wrote:
> On Sun, Jun 21, 2009 at 11:21:00PM -0700, Chris Anderson wrote:
>> My gut reaction is that normalizing strings using NFC [1] is not appropriate
>> for a database. Here's why we should treat strings as binary and not worry
>> about unicode normalization at all:
> [...]
>> First of all, I'm certain we can't require that all input already be NFC
>> normalized.
> [...]
>> Secondly, we're a database, so I find highly suspicious the notion that we
>> should auto-normalize user input on-the-quiet.
> [...]
>> So we can't require normalized input and we can't auto-normalize.
>
> CouchDB would create a canonicalised copy of the document while creating the
> document hash. There is no reason why CouchDB, or the clients, should worry
> about canonicalising the actual documents.
>
>> Where does this leave us?
>
> Canonicalisation is a temporary step, so there are no problems.
>
Works for me, then. Hadn't really considered that we won't be saving
the canonicalized versions, just using them as input to the hash
function.
My larger point is that normalization is basically an optimization.
Occasionally getting the hash wrong (for whatever reason) is just
going to result in spurious conflicts, which aren't critical errors,
just an annoyance as the application will sometimes have to repair the
conflicts. Presumably you'll already be repairing conflicts, so fixing
ones that result from spurious conflicts (especially as they are so
rare) is not a big cost, and should be easy as the application can
just pick a random version of the doc, with no ill effects.
I think this makes the NFC stuff a nice-to-have, not a necessity.
Chris
>> > Unicode normalisation is an issue for clients because it requires they have
>> > access to a Unicode NFC function.
>
> Why would clients need to worry about this? CouchDB is creating the hashes.
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Sun, Jun 21, 2009 at 11:21:00PM -0700, Chris Anderson wrote:
> My gut reaction is that normalizing strings using NFC [1] is not appropriate
> for a database. Here's why we should treat strings as binary and not worry
> about unicode normalization at all:
[...]
> First of all, I'm certain we can't require that all input already be NFC
> normalized.
[...]
> Secondly, we're a database, so I find highly suspicious the notion that we
> should auto-normalize user input on-the-quiet.
[...]
> So we can't require normalized input and we can't auto-normalize.
CouchDB would create a canonicalised copy of the document while creating the
document hash. There is no reason why CouchDB, or the clients, should worry
about canonicalising the actual documents.
> Where does this leave us?
Canonicalisation is a temporary step, so there are no problems.
> > Unicode normalisation is an issue for clients because it requires they have
> > access to a Unicode NFC function.
Why would clients need to worry about this? CouchDB is creating the hashes.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:46 AM, Damien Katz<da...@apache.org> wrote:
>
> On Jun 30, 2009, at 11:22 AM, Noah Slater wrote:
>
>> On Tue, Jun 30, 2009 at 07:12:07AM -0400, Damien Katz wrote:
>>>
>>> Im not sure I understand why we can't just calculate and send the MD5
>>> header for the content range.
>>
>> We could, but are you not proposing that we use this value for the
>> document
>> revision? If that is the case, when you do range requests, the hash sent
>> back
>> doesn't actually correspond to anything. If I used the hash from the final
>> range
>> request of a document to post an update, it would presumably fail.
>
> To clarify, the point of deterministic rev ids is only to avoid unnecessary
> conflicts when the identical edits are made on 2 different replicas. If the
> content was identical when editing the same revision, it should not be a
> conflict. If we had a canonical representation of the document, we could
> also use the determanistic rev ids for integrity checking, but we don't have
> a canonical representation, and creating one is very difficult to get right.
>
We most definitely do *not* need a format canonical JSON RFC to do
deterministic document revisions that can be used for integrity
checking. The only thing a formal canonical specification allows is
for non-erlang clients to compute the deterministic revisions.
A side benefit of a formal canonicalization is that if it included
unicode and float normalization we would then be removing the
possibility of spurious conflicts that are the result of differences
in information *representation*. It's important to note that
non-normalized algorithms would still be quite capable of avoiding
many spurious conflicts, normalization just makes the chance of
spurious conflict (theoretically) zero.
In other words, we're more than free to just md5 the
term_to_binary(Json) and use that as a deterministic revision. Things
like sorting object members and normalization are just optimizations
to reduce conflicts when the information is constant but the
representation has changed.
Paul Davis
> What I'm proposing is that we only use content-MD5 for payload integrity
> checking. It will not being used for security and it cannot be validated
> against the rev id because they will always be different. The rev Id will be
> generated based on the erlang term format of the document, not the UTF8 JSON
> string that gets sent to the client.
>
> So the server will send it's responses (perhaps optionally) with a MD5 hash
> to detect packet corruption. Clients, when they send docs and attachments,
> can send the payload with a content-MD5 header and the server will check it
> to make sure it's uncorrupted. As it writes the data to disk the server will
> compute the MD5 hash, for it's own integrity checking later.
>
> So for example, the replicator will check the md5 sig from the server and
> send it's own md5 sig when writing data. This prevents network problems from
> introducing corruptions to data as it replicates.
>
> -Damien
>
>>
>> Best,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
On Jun 30, 2009, at 11:22 AM, Noah Slater wrote:
> On Tue, Jun 30, 2009 at 07:12:07AM -0400, Damien Katz wrote:
>> Im not sure I understand why we can't just calculate and send the MD5
>> header for the content range.
>
> We could, but are you not proposing that we use this value for the
> document
> revision? If that is the case, when you do range requests, the hash
> sent back
> doesn't actually correspond to anything. If I used the hash from the
> final range
> request of a document to post an update, it would presumably fail.
To clarify, the point of deterministic rev ids is only to avoid
unnecessary conflicts when the identical edits are made on 2 different
replicas. If the content was identical when editing the same revision,
it should not be a conflict. If we had a canonical representation of
the document, we could also use the determanistic rev ids for
integrity checking, but we don't have a canonical representation, and
creating one is very difficult to get right.
What I'm proposing is that we only use content-MD5 for payload
integrity checking. It will not being used for security and it cannot
be validated against the rev id because they will always be different.
The rev Id will be generated based on the erlang term format of the
document, not the UTF8 JSON string that gets sent to the client.
So the server will send it's responses (perhaps optionally) with a MD5
hash to detect packet corruption. Clients, when they send docs and
attachments, can send the payload with a content-MD5 header and the
server will check it to make sure it's uncorrupted. As it writes the
data to disk the server will compute the MD5 hash, for it's own
integrity checking later.
So for example, the replicator will check the md5 sig from the server
and send it's own md5 sig when writing data. This prevents network
problems from introducing corruptions to data as it replicates.
-Damien
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:43 AM, Noah Slater<ns...@apache.org> wrote:
> On Tue, Jun 30, 2009 at 11:38:51AM -0400, Paul Davis wrote:
>> As I read the original email about Content-MD5 is that they are
>> specifically *not* for document revisions because then we'd have to do
>> canonicalization.
>
> I thought the whole point was to replace the current revision code with MD5
> hashes so that the document revisions are deterministic. If we cant do this,
> what is the point of this whole exercise?
>
>> But the RFC specifically states that the Content-MD5 is applied to the
>> canonical version.
>
> This tripped me up at first too. The RFC is talking about the MIME canonical
> version. If you re-read, you'll see that this is about newline normalisation,
> and other sundry details of the transport encoding.
>
Oh sweet. I kept reading "model of MIME" to mean something like
"canonical version according the content-type".
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Tue, Jun 30, 2009 at 11:38:51AM -0400, Paul Davis wrote:
> As I read the original email about Content-MD5 is that they are
> specifically *not* for document revisions because then we'd have to do
> canonicalization.
I thought the whole point was to replace the current revision code with MD5
hashes so that the document revisions are deterministic. If we cant do this,
what is the point of this whole exercise?
> But the RFC specifically states that the Content-MD5 is applied to the
> canonical version.
This tripped me up at first too. The RFC is talking about the MIME canonical
version. If you re-read, you'll see that this is about newline normalisation,
and other sundry details of the transport encoding.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:22 AM, Noah Slater<ns...@apache.org> wrote:
> On Tue, Jun 30, 2009 at 07:12:07AM -0400, Damien Katz wrote:
>> Im not sure I understand why we can't just calculate and send the MD5
>> header for the content range.
>
> We could, but are you not proposing that we use this value for the document
> revision?
As I read the original email about Content-MD5 is that they are
specifically *not* for document revisions because then we'd have to do
canonicalization. But the RFC specifically states that the Content-MD5
is applied to the canonical version.
> If that is the case, when you do range requests, the hash sent back
> doesn't actually correspond to anything. If I used the hash from the final range
> request of a document to post an update, it would presumably fail.
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Tue, Jun 30, 2009 at 07:12:07AM -0400, Damien Katz wrote:
> Im not sure I understand why we can't just calculate and send the MD5
> header for the content range.
We could, but are you not proposing that we use this value for the document
revision? If that is the case, when you do range requests, the hash sent back
doesn't actually correspond to anything. If I used the hash from the final range
request of a document to post an update, it would presumably fail.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Tue, Jun 30, 2009 at 12:21:19PM -0400, Paul Davis wrote:
> On Tue, Jun 30, 2009 at 11:23 AM, Noah Slater<ns...@apache.org> wrote:
> > On Tue, Jun 30, 2009 at 10:44:51AM -0400, Paul Davis wrote:
>
> Technically there are HTTP Footers [1], but i've never actually seen
> them used [2].
Interesting. Might be good for streaming of documents with Content-MD5.
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:23 AM, Noah Slater<ns...@apache.org> wrote:
> On Tue, Jun 30, 2009 at 10:44:51AM -0400, Paul Davis wrote:
Technically there are HTTP Footers [1], but i've never actually seen
them used [2].
>
> [citation needed]
>
[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
[2] Personal communication
Though they're called trailers.
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:23 AM, Noah Slater<ns...@apache.org> wrote:
> On Tue, Jun 30, 2009 at 10:44:51AM -0400, Paul Davis wrote:
>> Technically there are HTTP Footers, but i've never actually seen them used.
>
> [citation needed]
>
Touché. It looks like I was pulling in memories of MIME stuff. Still
can't find an RFC that describes them though.
> --
> Noah Slater, http://tumbolia.org/nslater
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Tue, Jun 30, 2009 at 10:44:51AM -0400, Paul Davis wrote:
> Technically there are HTTP Footers, but i've never actually seen them used.
[citation needed]
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 11:04 AM, Damien Katz<da...@apache.org> wrote:
> Definitely don't buffer the whole thing. Footers are the way to go for
> efficiency reasons, but if there is a problem supporting them the
> alternative is to just read over the section of file and compute the MD5
> hash, then read it again and stream it to the client. It sounds more
> expensive to do it that way, but double reading eliminates double caching as
> file system cache will keep it in memory most of the time anyway. And the FS
> cache is smarter about not swapping out more important data if the section
> is large.
>
This works fine when the range isn't expensive to calculate.
In the end I'd agree that prohibiting an MD5 purely because its a
range request seems silly when it should more likely be an
implementation decision.
> But footers are definitely the way to go for efficiency. Which brings up a
> good question, are there known problems with footer support?
>
The only problem I can think of is that I can't think of anyone that
supports them. That could just be because I've also never seen
anything that actually sends them and thus not had to look for client
support.
> -Damien
>
>
> On Jun 30, 2009, at 10:44 AM, Paul Davis wrote:
>
>> On Tue, Jun 30, 2009 at 7:12 AM, Damien Katz<da...@apache.org> wrote:
>>>
>>> On Jun 30, 2009, at 12:17 AM, Noah Slater wrote:
>>>
>>>> On Fri, Jun 26, 2009 at 07:08:32AM -0400, Damien Katz wrote:
>>>>>
>>>>> Md5 here is for integrity purposes, not security, so manufactured
>>>>> collisions aren't a problem we are worried about. And I don't think
>>>>> there is standard SHA1 header, not that I could find anyway.
>>>>
>>>> I've been seeing some unrelated emails go past on the W3C HTTP WG
>>>> mailing
>>>> list
>>>> about Content-MD5 header which reminded me of this thread. It seems that
>>>> this
>>>> value must be calculated from the MIME canonical response body, which
>>>> means a
>>>> different value for content ranges. This presumably means that CouchDB
>>>> must
>>>> refuse content range requests, send an MD5 value that does not match the
>>>> document revision, or break RFC 1864.
>>>
>>> Im not sure I understand why we can't just calculate and send the MD5
>>> header
>>> for the content range.
>>>
>>
>> I reckon you'd have to buffer the response no? Hard to know the MD5 of
>> an a priori unknown set of bytes until the end of the range which
>> kinda conflicts with sending the MD5 as a header. Technically there
>> are HTTP Footers, but i've never actually seen them used.
>>
>>> -Damien
>>>
>>>>
>>>> Best,
>>>>
>>>> --
>>>> Noah Slater, http://tumbolia.org/nslater
>>>
>>>
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
Definitely don't buffer the whole thing. Footers are the way to go for
efficiency reasons, but if there is a problem supporting them the
alternative is to just read over the section of file and compute the
MD5 hash, then read it again and stream it to the client. It sounds
more expensive to do it that way, but double reading eliminates double
caching as file system cache will keep it in memory most of the time
anyway. And the FS cache is smarter about not swapping out more
important data if the section is large.
But footers are definitely the way to go for efficiency. Which brings
up a good question, are there known problems with footer support?
-Damien
On Jun 30, 2009, at 10:44 AM, Paul Davis wrote:
> On Tue, Jun 30, 2009 at 7:12 AM, Damien Katz<da...@apache.org> wrote:
>>
>> On Jun 30, 2009, at 12:17 AM, Noah Slater wrote:
>>
>>> On Fri, Jun 26, 2009 at 07:08:32AM -0400, Damien Katz wrote:
>>>>
>>>> Md5 here is for integrity purposes, not security, so manufactured
>>>> collisions aren't a problem we are worried about. And I don't think
>>>> there is standard SHA1 header, not that I could find anyway.
>>>
>>> I've been seeing some unrelated emails go past on the W3C HTTP WG
>>> mailing
>>> list
>>> about Content-MD5 header which reminded me of this thread. It
>>> seems that
>>> this
>>> value must be calculated from the MIME canonical response body,
>>> which
>>> means a
>>> different value for content ranges. This presumably means that
>>> CouchDB
>>> must
>>> refuse content range requests, send an MD5 value that does not
>>> match the
>>> document revision, or break RFC 1864.
>>
>> Im not sure I understand why we can't just calculate and send the
>> MD5 header
>> for the content range.
>>
>
> I reckon you'd have to buffer the response no? Hard to know the MD5 of
> an a priori unknown set of bytes until the end of the range which
> kinda conflicts with sending the MD5 as a header. Technically there
> are HTTP Footers, but i've never actually seen them used.
>
>> -Damien
>>
>>>
>>> Best,
>>>
>>> --
>>> Noah Slater, http://tumbolia.org/nslater
>>
>>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Tue, Jun 30, 2009 at 7:12 AM, Damien Katz<da...@apache.org> wrote:
>
> On Jun 30, 2009, at 12:17 AM, Noah Slater wrote:
>
>> On Fri, Jun 26, 2009 at 07:08:32AM -0400, Damien Katz wrote:
>>>
>>> Md5 here is for integrity purposes, not security, so manufactured
>>> collisions aren't a problem we are worried about. And I don't think
>>> there is standard SHA1 header, not that I could find anyway.
>>
>> I've been seeing some unrelated emails go past on the W3C HTTP WG mailing
>> list
>> about Content-MD5 header which reminded me of this thread. It seems that
>> this
>> value must be calculated from the MIME canonical response body, which
>> means a
>> different value for content ranges. This presumably means that CouchDB
>> must
>> refuse content range requests, send an MD5 value that does not match the
>> document revision, or break RFC 1864.
>
> Im not sure I understand why we can't just calculate and send the MD5 header
> for the content range.
>
I reckon you'd have to buffer the response no? Hard to know the MD5 of
an a priori unknown set of bytes until the end of the range which
kinda conflicts with sending the MD5 as a header. Technically there
are HTTP Footers, but i've never actually seen them used.
> -Damien
>
>>
>> Best,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
On Jun 30, 2009, at 12:17 AM, Noah Slater wrote:
> On Fri, Jun 26, 2009 at 07:08:32AM -0400, Damien Katz wrote:
>> Md5 here is for integrity purposes, not security, so manufactured
>> collisions aren't a problem we are worried about. And I don't think
>> there is standard SHA1 header, not that I could find anyway.
>
> I've been seeing some unrelated emails go past on the W3C HTTP WG
> mailing list
> about Content-MD5 header which reminded me of this thread. It seems
> that this
> value must be calculated from the MIME canonical response body,
> which means a
> different value for content ranges. This presumably means that
> CouchDB must
> refuse content range requests, send an MD5 value that does not match
> the
> document revision, or break RFC 1864.
Im not sure I understand why we can't just calculate and send the MD5
header for the content range.
-Damien
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Fri, Jun 26, 2009 at 07:08:32AM -0400, Damien Katz wrote:
> Md5 here is for integrity purposes, not security, so manufactured
> collisions aren't a problem we are worried about. And I don't think
> there is standard SHA1 header, not that I could find anyway.
I've been seeing some unrelated emails go past on the W3C HTTP WG mailing list
about Content-MD5 header which reminded me of this thread. It seems that this
value must be calculated from the MIME canonical response body, which means a
different value for content ranges. This presumably means that CouchDB must
refuse content range requests, send an MD5 value that does not match the
document revision, or break RFC 1864.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
Md5 here is for integrity purposes, not security, so manufactured
collisions aren't a problem we are worried about. And I don't think
there is standard SHA1 header, not that I could find anyway.
-Damien
On Jun 26, 2009, at 1:32 AM, kowsik wrote:
> Please use SHA-1 because creating collisions with MD5 is trivial:
>
> http://web.archive.org/web/20070604205756/http://www.infosec.sdu.edu.cn/paper/md5-attack.pdf
> http://www.mscs.dal.ca/~selinger/md5collision/
>
> etc.
>
> Google for "md5 collision". Effectively, what this means that it's
> easy to generate two documents that have the same MD5 hash. I'm sure
> SHA-1 will be an issue at "some point in the future", but MD5 is
> already broken from a hashing perspective.
>
> K.
>
> On Thu, Jun 25, 2009 at 2:37 PM, Damien Katz<da...@apache.org> wrote:
>> I am now working on an implementation of deterministic revs. After
>> a lot of
>> thinking about this, I've decided to not reuse the revision ids for
>> integrity checking. The canonicalization problem is unresolved and
>> using a
>> CouchDB specific canonicalization means other libs/langs/platforms
>> can't
>> play easily with CouchDB replication.
>>
>> Integrity will be preserved by use of Content-MD5 when
>> transferring/replicating documents, and checking the document
>> hashing when
>> reading from of disk. The replicator http client will check the
>> integrity of
>> the network bodies.
>>
>> If you need end-to-end integrity checking, you can use an application
>> specific scheme to sign/hash various fields and attachments, if you
>> can deal
>> with the string and floating point canonicalization issues.
>>
>> My plan is that when generating new rev ids, CouchDB will
>> deterministically
>> generate the same revision id when edited with the same data. But
>> it still
>> is specific to the version of CouchDB and it's dependencies
>> (version of
>> Erlang, version of ICU, etc). It usually be the same across
>> versions, but is
>> not guaranteed.
>>
>> What this will allow is for a single client to send the same edits
>> to 2
>> identical Erlang servers and see the same revids generated on both.
>> Optionally will allow that if 2 clients make byte identical saves
>> for a
>> document, they will get the same revision, and you don't need to
>> return a
>> conflict error the second client to save. I'm not sure about
>> implementing
>> this though.
>>
>> To implement this couchdb will store a md5 hash of the all the
>> attachments
>> along with the json document, when saving a new document we hash
>> the native
>> document and the attachment hashes together to generate the
>> revision id.
>>
>> CouchDB will also store a md5 hash of the json document itself.
>> This will
>> give us disk integrity checking for all documents and their
>> attachments in a
>> database. When CouchdB encounters a corrupt document or attachment
>> it will
>> stop what it's doing and return an error. The admin can restore
>> from backup
>> or recreate by deleting and re-replicating from a peer.
>>
>> I think this is the most pragmatic way to do deterministic revs and
>> integrity checking. That is, do as little as possible and let
>> others deal
>> with the problems and implications of canonicalization if they want
>> to to do
>> end to end integrity checking.
>>
>> Feedback please.
>>
>> -Damien
>>
>>>
>>
>>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by kowsik <ko...@gmail.com>.
Please use SHA-1 because creating collisions with MD5 is trivial:
http://web.archive.org/web/20070604205756/http://www.infosec.sdu.edu.cn/paper/md5-attack.pdf
http://www.mscs.dal.ca/~selinger/md5collision/
etc.
Google for "md5 collision". Effectively, what this means that it's
easy to generate two documents that have the same MD5 hash. I'm sure
SHA-1 will be an issue at "some point in the future", but MD5 is
already broken from a hashing perspective.
K.
On Thu, Jun 25, 2009 at 2:37 PM, Damien Katz<da...@apache.org> wrote:
> I am now working on an implementation of deterministic revs. After a lot of
> thinking about this, I've decided to not reuse the revision ids for
> integrity checking. The canonicalization problem is unresolved and using a
> CouchDB specific canonicalization means other libs/langs/platforms can't
> play easily with CouchDB replication.
>
> Integrity will be preserved by use of Content-MD5 when
> transferring/replicating documents, and checking the document hashing when
> reading from of disk. The replicator http client will check the integrity of
> the network bodies.
>
> If you need end-to-end integrity checking, you can use an application
> specific scheme to sign/hash various fields and attachments, if you can deal
> with the string and floating point canonicalization issues.
>
> My plan is that when generating new rev ids, CouchDB will deterministically
> generate the same revision id when edited with the same data. But it still
> is specific to the version of CouchDB and it's dependencies (version of
> Erlang, version of ICU, etc). It usually be the same across versions, but is
> not guaranteed.
>
> What this will allow is for a single client to send the same edits to 2
> identical Erlang servers and see the same revids generated on both.
> Optionally will allow that if 2 clients make byte identical saves for a
> document, they will get the same revision, and you don't need to return a
> conflict error the second client to save. I'm not sure about implementing
> this though.
>
> To implement this couchdb will store a md5 hash of the all the attachments
> along with the json document, when saving a new document we hash the native
> document and the attachment hashes together to generate the revision id.
>
> CouchDB will also store a md5 hash of the json document itself. This will
> give us disk integrity checking for all documents and their attachments in a
> database. When CouchdB encounters a corrupt document or attachment it will
> stop what it's doing and return an error. The admin can restore from backup
> or recreate by deleting and re-replicating from a peer.
>
> I think this is the most pragmatic way to do deterministic revs and
> integrity checking. That is, do as little as possible and let others deal
> with the problems and implications of canonicalization if they want to to do
> end to end integrity checking.
>
> Feedback please.
>
> -Damien
>
>>
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Freddy Bowen <fr...@gmail.com>.
On Fri, Jun 26, 2009 at 7:21 AM, Damien Katz <da...@apache.org> wrote:
>
>
>> But it still is specific to the version of CouchDB and it's dependencies
>>> (version of Erlang, version of ICU, etc). It usually be the same across
>>> versions, but is not guaranteed.
>>>
>>
>> If we're doing content hashing, why would this matter?
>>
>
> Because we don't have a formal canonical format, so we aren't even trying.
> We'll be hashing whatever representation we have in-memory, and that could
> change version to version.
It concerns me that replicating partners with a differing versions of
CouchDB or any dependencies (ie. minor version differences?) could generate
different rev hashes of identical content. I am ignorant as to how
different versions would actually affect in-memory representations so I
don't know if this is actually a concern or more like a footnoted
possibility. Does this strike anybody else as potentially bad?
-FB
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Chris Anderson <jc...@apache.org>.
On Fri, Jun 26, 2009 at 4:21 AM, Damien Katz<da...@apache.org> wrote:
>
> On Jun 25, 2009, at 6:53 PM, Noah Slater wrote:
>
>> On Thu, Jun 25, 2009 at 05:37:21PM -0400, Damien Katz wrote:
>>>
>>> Integrity will be preserved by use of Content-MD5
>>
>> Bike shed: what about the stronger SHA family of hashes?
>
> Content-MD5 is standard header, I can find no others headers to do integrity
> hashing.
>>
>>> But it still is specific to the version of CouchDB and it's dependencies
>>> (version of Erlang, version of ICU, etc). It usually be the same across
>>> versions, but is not guaranteed.
>>
>> If we're doing content hashing, why would this matter?
>
> Because we don't have a formal canonical format, so we aren't even trying.
> We'll be hashing whatever representation we have in-memory, and that could
> change version to version.
>
>>
>>> Optionally will allow that if 2 clients make byte identical saves for a
>>> document, they will get the same revision, and you don't need to return a
>>> conflict error the second client to save.
>>
>> Are there any security issues around possible hash collisions?
>
> No, we aren't checking them later.
>
This all sounds very sensible to me.
On the security note: the only time I can see a manufactured hash
collision mattering is if you know that 2 nodes will eventually
replicate, and you provide a poison version of a document to one of
them. In that case what would be the behavior?
Pragmatically, it might be worth checking to see if most client
(firefox, ruby, etc) will save the same document with different binary
representations to 2 different servers.
The easiest way to fix this, without going all the through the looking
glass, would be to recursively sort any eJSON proplists we encounter
in the #doc.body. This way at least the most common case of
should-be-the-same-but-isn't would be protected against, without
adding much weight to the implementation.
The argument against this is that it's a slippery slope. If we don't
slide down it, it's not bad. If we slide, then we end up in the
canonicalization muck, so let's not slide. If we think we can't help
but slide, then forget I mentioned it, and put the onus on clients to
provide the same binary data when they want the same rev.
Overall the punting is in the right place, Damien. Good call.
>>
>>> I think this is the most pragmatic way to do deterministic revs and
>>> integrity
>>> checking. That is, do as little as possible and let others deal with the
>>> problems and implications of canonicalization if they want to to do end
>>> to end
>>> integrity checking.
>>
>> Seems like a reasonable approach to me.
>>
>> Best,
>>
>> --
>> Noah Slater, http://tumbolia.org/nslater
>
>
--
Chris Anderson
http://jchrisa.net
http://couch.io
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
On Jun 25, 2009, at 6:53 PM, Noah Slater wrote:
> On Thu, Jun 25, 2009 at 05:37:21PM -0400, Damien Katz wrote:
>> Integrity will be preserved by use of Content-MD5
>
> Bike shed: what about the stronger SHA family of hashes?
Content-MD5 is standard header, I can find no others headers to do
integrity hashing.
>
>> But it still is specific to the version of CouchDB and it's
>> dependencies
>> (version of Erlang, version of ICU, etc). It usually be the same
>> across
>> versions, but is not guaranteed.
>
> If we're doing content hashing, why would this matter?
Because we don't have a formal canonical format, so we aren't even
trying. We'll be hashing whatever representation we have in-memory,
and that could change version to version.
>
>> Optionally will allow that if 2 clients make byte identical saves
>> for a
>> document, they will get the same revision, and you don't need to
>> return a
>> conflict error the second client to save.
>
> Are there any security issues around possible hash collisions?
No, we aren't checking them later.
>
>> I think this is the most pragmatic way to do deterministic revs and
>> integrity
>> checking. That is, do as little as possible and let others deal
>> with the
>> problems and implications of canonicalization if they want to to do
>> end to end
>> integrity checking.
>
> Seems like a reasonable approach to me.
>
> Best,
>
> --
> Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Thu, Jun 25, 2009 at 05:37:21PM -0400, Damien Katz wrote:
> Integrity will be preserved by use of Content-MD5
Bike shed: what about the stronger SHA family of hashes?
> But it still is specific to the version of CouchDB and it's dependencies
> (version of Erlang, version of ICU, etc). It usually be the same across
> versions, but is not guaranteed.
If we're doing content hashing, why would this matter?
> Optionally will allow that if 2 clients make byte identical saves for a
> document, they will get the same revision, and you don't need to return a
> conflict error the second client to save.
Are there any security issues around possible hash collisions?
> I think this is the most pragmatic way to do deterministic revs and integrity
> checking. That is, do as little as possible and let others deal with the
> problems and implications of canonicalization if they want to to do end to end
> integrity checking.
Seems like a reasonable approach to me.
Best,
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Paul Davis <pa...@gmail.com>.
On Thu, Jun 25, 2009 at 5:37 PM, Damien Katz<da...@apache.org> wrote:
> I am now working on an implementation of deterministic revs. After a lot of
> thinking about this, I've decided to not reuse the revision ids for
> integrity checking. The canonicalization problem is unresolved and using a
> CouchDB specific canonicalization means other libs/langs/platforms can't
> play easily with CouchDB replication.
>
> Integrity will be preserved by use of Content-MD5 when
> transferring/replicating documents, and checking the document hashing when
> reading from of disk. The replicator http client will check the integrity of
> the network bodies.
>
> If you need end-to-end integrity checking, you can use an application
> specific scheme to sign/hash various fields and attachments, if you can deal
> with the string and floating point canonicalization issues.
>
> My plan is that when generating new rev ids, CouchDB will deterministically
> generate the same revision id when edited with the same data. But it still
> is specific to the version of CouchDB and it's dependencies (version of
> Erlang, version of ICU, etc). It usually be the same across versions, but is
> not guaranteed.
>
> What this will allow is for a single client to send the same edits to 2
> identical Erlang servers and see the same revids generated on both.
> Optionally will allow that if 2 clients make byte identical saves for a
> document, they will get the same revision, and you don't need to return a
> conflict error the second client to save. I'm not sure about implementing
> this though.
>
> To implement this couchdb will store a md5 hash of the all the attachments
> along with the json document, when saving a new document we hash the native
> document and the attachment hashes together to generate the revision id.
>
> CouchDB will also store a md5 hash of the json document itself. This will
> give us disk integrity checking for all documents and their attachments in a
> database. When CouchdB encounters a corrupt document or attachment it will
> stop what it's doing and return an error. The admin can restore from backup
> or recreate by deleting and re-replicating from a peer.
>
> I think this is the most pragmatic way to do deterministic revs and
> integrity checking. That is, do as little as possible and let others deal
> with the problems and implications of canonicalization if they want to to do
> end to end integrity checking.
>
> Feedback please.
>
One thing that strikes me as potentially bad is that the signature
can't be recalculated. Not sure if that's important or not.
> -Damien
>
>>
>
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Damien Katz <da...@apache.org>.
I am now working on an implementation of deterministic revs. After a
lot of thinking about this, I've decided to not reuse the revision ids
for integrity checking. The canonicalization problem is unresolved and
using a CouchDB specific canonicalization means other libs/langs/
platforms can't play easily with CouchDB replication.
Integrity will be preserved by use of Content-MD5 when transferring/
replicating documents, and checking the document hashing when reading
from of disk. The replicator http client will check the integrity of
the network bodies.
If you need end-to-end integrity checking, you can use an application
specific scheme to sign/hash various fields and attachments, if you
can deal with the string and floating point canonicalization issues.
My plan is that when generating new rev ids, CouchDB will
deterministically generate the same revision id when edited with the
same data. But it still is specific to the version of CouchDB and it's
dependencies (version of Erlang, version of ICU, etc). It usually be
the same across versions, but is not guaranteed.
What this will allow is for a single client to send the same edits to
2 identical Erlang servers and see the same revids generated on both.
Optionally will allow that if 2 clients make byte identical saves for
a document, they will get the same revision, and you don't need to
return a conflict error the second client to save. I'm not sure about
implementing this though.
To implement this couchdb will store a md5 hash of the all the
attachments along with the json document, when saving a new document
we hash the native document and the attachment hashes together to
generate the revision id.
CouchDB will also store a md5 hash of the json document itself. This
will give us disk integrity checking for all documents and their
attachments in a database. When CouchdB encounters a corrupt document
or attachment it will stop what it's doing and return an error. The
admin can restore from backup or recreate by deleting and re-
replicating from a peer.
I think this is the most pragmatic way to do deterministic revs and
integrity checking. That is, do as little as possible and let others
deal with the problems and implications of canonicalization if they
want to to do end to end integrity checking.
Feedback please.
-Damien
>
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Brian Candler <B....@pobox.com>.
On Tue, Jun 23, 2009 at 05:46:52PM +0100, Noah Slater wrote:
> On Tue, Jun 23, 2009 at 09:26:23AM +0100, Brian Candler wrote:
> > As an aside: I support that subtly different encodings of the "same"
> > document (according to NFC) should have different revs, because (a) it's
> > unlikely that multiple different client implementations will be making the
> > same changes to the same documents (i.e. the clients in a cluster are likely
> > to be homogeneous), and (b) such conflicts are easy to resolve anyway.
>
> Clients on the Web are anything but homogeneous!
True. I was thinking more of the three-tier architecture, where the middle
layer is scaled horizontally, and consists of identical nodes.
For the two-tier "Couchapp" type application, where each client is a
completely independent user, it seems less likely to me that two different
users would make exactly the same overlapping edit to the same document. Not
impossible of course. Maybe some Couchapps will have extensive
conflict-resolution algorithms written in Javascript and running inside
browsers. I'd hate to have to do that myself :-)
The idea of fully-trusted clients, with all business logic running in the
client, scares me. You may as well just give raw SQL connections to anyone
who wants them. I've seen client apps written that way too of course :-)
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Noah Slater <ns...@apache.org>.
On Tue, Jun 23, 2009 at 09:26:23AM +0100, Brian Candler wrote:
> As an aside: I support that subtly different encodings of the "same"
> document (according to NFC) should have different revs, because (a) it's
> unlikely that multiple different client implementations will be making the
> same changes to the same documents (i.e. the clients in a cluster are likely
> to be homogeneous), and (b) such conflicts are easy to resolve anyway.
Clients on the Web are anything but homogeneous!
--
Noah Slater, http://tumbolia.org/nslater
Re: Unicode normalization (was Re: The 1.0 Thread)
Posted by Brian Candler <B....@pobox.com>.
On Sun, Jun 21, 2009 at 11:21:00PM -0700, Chris Anderson wrote:
> A normal
> user is not going to understand the first bit of the fact that the
> underlying binary representation of their text could be subtly
> different in a way that would be invisible to them.
...
> Secondly, we're a database, so I find highly suspicious the notion
> that we should auto-normalize user input on-the-quiet.
Then maybe is it worth going the whole hog, and just storing the received
JSON directly to disk as a string? This takes out the JSON->erlang parsing
when storing documents, and the erlang->JSON serialisation when sending on
to the view server, or when retrieving documents for the client.
Of course, there is metadata which CouchDB adds, like _id, _rev etc. This
could be stored separately alongside the document, and then shoehorned in
when you retrieve the document (e.g. as simple as inserting some text after
the initial '{').
This gives some interesting future options: e.g. moving the metadata into
HTTP headers, at which point there is no requirement for the document to be
in JSON form at all. It just has to be in some format that the view server
is happy to parse.
As an aside: I support that subtly different encodings of the "same"
document (according to NFC) should have different revs, because (a) it's
unlikely that multiple different client implementations will be making the
same changes to the same documents (i.e. the clients in a cluster are likely
to be homogeneous), and (b) such conflicts are easy to resolve anyway.
Furthermore:
> "don't mutilate strings you didn't edit" so as long as client software
> doesn't go jiggling forms to other random look-alike codepoints
> without asking, any potential trouble is confined to fields actually
> effected by an update.
It's probably not reasonable to make this requirement. Most client software
will deserialise JSON into some internal form (a Ruby hash, a Python dict,
or whatever), at which point transformations will take place, so turning it
back into JSON may well not give exactly the same serialisation. Ruby 1.8
won't even maintain the member ordering.