CouchDB 2.0: breaking the backward compatibility

[Alexander Shorin <kx...@gmail.com>]

Hi devs,

CouchDB 2.0 obliviously breaks the backward compatibility. I think
this is good moment to review our current issues which are also
requires to break the BC.

For instance, spidermonkey version. We cannot use recent SM versions
since anonymous functions are gone after 1.8.5 release and that moment
breaks all the existed design functions.
https://issues.apache.org/jira/browse/COUCHDB-1397

Another one is about changing Revision (_rev) into something more less
misleading:
https://issues.apache.org/jira/browse/COUCHDB-1399
(or we eventually have to implement version control module for CouchDB one day)

Our view server is compiles functions on each view index update
instead of reusing inner cache. This is because of out-dated protocol:
others design function are works differently from views. While it's
good to change and improve query server protocol completely, this task
requires more time to be done. We should have a least plan B to do
small steps in good direction.

API consistency:
- timeout and heartbeat params for /_db_updates works in different way
then the same parameters for changes feed;
- we need to find the way to pass open_revs in POST body instead of
tweaking max URL param;
- we have /db/_revs_diff and /db/_revs_missing endpoints which are
doing the same job. Well, the latter is only used for pre-1.1 CouchDB
replicator.
- /db/doc accepts conflicts, deleted_conflicts and revs params. In the
same time we provides meta one which includes each of specified.
- make eventsource feed to follow the specification format more better
then it does now
- MVCC for /db/_security and allow atomic changes for admins/members only

That's only a few of the moments which are on my mind now, I believe
there are a lot of others. While each of them shifts 2.0 release more
in time, we shouldn't miss the moment when we can break the backward
compatibility for free to improve CouchDB and cleanup legacy bits.

I propose to collect all the BC stuff on JIRA. After that we could
decide what we could change, what we will and what we'll not (because
of no reasons for or lack of time).

--
,,,^..^,,,

Re: CouchDB 2.0: breaking the backward compatibility

[Alexander Shorin <kx...@gmail.com>]

I'm not sure that we can handle whole protocol redesign till 2.0
release too, but at least we could try to fix the flaws of the current
one. In case if this will bring some changes which users may notice,
for sure (e.g. break all the query servers just for 5% speedup might
be bad idea).
--
,,,^..^,,,


On Sun, Jul 13, 2014 at 11:31 PM, Joan Touzet <wo...@apache.org> wrote:
> Improving the view server protocol is a great idea, but it is appropriate
> for a 2.0 timeframe? I would think it would make more sense in a 3.0
> timeframe, given 2.0 is all about merging forks, not writing new features
> entirely from scratch.
>
> -Joan
>
> ----- Original Message -----
> From: "Robert Samuel Newson" <rn...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Sunday, July 13, 2014 8:52:40 AM
> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>
>
> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>
> A better view server protocol is also a great idea.
>
>
> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>
>>
>> On 13/07/14 23:47, Alexander Shorin wrote:
>>> Our view server is compiles functions on each view index update
>>> instead of reusing inner cache. This is because of out-dated protocol:
>>> others design function are works differently from views. While it's
>>> good to change and improve query server protocol completely, this task
>>> requires more time to be done. We should have a least plan B to do
>>> small steps in good direction.
>> As already suggested, here is my proposal for 2.0 view/query server:
>>
>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>
>> I welcome people to suggest improvements/changes/ideas.
>>
>> Kind regards,
>> Samuel
>

Re: CouchDB 2.0: breaking the backward compatibility

[Paul Davis <pa...@gmail.com>]

+1 to zlib/base64 strings as long as we make sure that the string
format is trivially and directly parseable (ie, no regex requirement
for parsing).

On Mon, Jul 14, 2014 at 4:30 PM, Robert Samuel Newson
<rn...@apache.org> wrote:
> Hrm, no, I think those would remain as non_neg_integer(), but the fate of single-node is not yet determined.
>
> B.
>
> On 14 Jul 2014, at 19:36, Joan Touzet <wo...@apache.org> wrote:
>
>> Yes to all 3 of your questions.
>>
>> Would we mandate this format even for single-node? We could "fake it" as if it was a q=1 db, i.e.
>>
>> "seq":"couchdb@foo,000-fff,12"
>>
>> If you want to allow replication from 1.x hosts you'd have to be more lenient ("should" not "must").
>>
>> ----- Original Message -----
>> From: "Robert Samuel Newson" <rn...@apache.org>
>> To: dev@couchdb.apache.org
>> Sent: Monday, July 14, 2014 10:34:21 AM
>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>
>>
>> Another thought occurs; BigCouch has a different format for update_seq that is notoriously ugly.
>>
>> We obviously need to encode more information for a sharded cluster (the update_seq of each shard, the range that it’s for and the node it resides on) but BigCouch also had to be compatible with CouchDB 1.x installations, so we add the sum of update sequences on the front, to trick the old replicator into working.
>>
>> While it is not necessary for a human to be able to read an update_seq value (aka: you should treat it as opaque JSON), it’s often useful to unpack these for diagnostic purposes. Our use of term_to_binary confounds non-erlang libraries from doing so.
>>
>> I propose we fix that in 2.0 which would require that the replicator checkpoints every N updates, and not when it sees the current update_seq exceed some delta from the update_seq of the last checkpoint.
>>
>> An example readable format would be;
>>
>> "seq":"couchdb@foo,000-ccc,12:couchdb@bar,d00-fff,10"
>>
>> that is, a formatted string.
>>
>> A few questions;
>>
>> 1) Should we obscure hostnames? (we could run then forward through sha1 or even pbkdf2, akin to how .known_hosts is protected by ssh)
>> 2) should we gzip encode the result?
>> 3) should we base64 the result?
>>
>> I think "yes" to all questions (and we would obviously have to base64 if we gzipped).
>>
>> Thoughts?
>>
>> B.
>>
>>
>> On 13 Jul 2014, at 21:23, Paul Davis <pa...@gmail.com> wrote:
>>
>>> Changing the default respones for conflicts to include all versions
>>> (or no version).
>>>
>>> Fix the list API (inside couchjs) so that its a pure callback like
>>> everything else. Not sure if we should necessarily completely revamp
>>> the whole query server protocol for 2.0. Given that its not user
>>> facing I'm less inclined to think it needs to be in a major release,
>>> ie we could add a new protocol in a minor release after 2.0.
>>>
>>> We should rename _rev to _mvcc (or _token or _lock or anything not
>>> _rev) finally.
>>>
>>> Removing all metadata from document bodies has been an oft requested change.
>>>
>>> Seems like there was a list of these things floating around a long time ago.
>>>
>>> On Sun, Jul 13, 2014 at 3:17 PM, Robert Samuel Newson
>>> <rn...@apache.org> wrote:
>>>>
>>>> Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.
>>>>
>>>> It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.
>>>>
>>>> With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.
>>>>
>>>> B.
>>>>
>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>
>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>> entirely from scratch.
>>>>>
>>>>> -Joan
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>> To: dev@couchdb.apache.org
>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>
>>>>>
>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>>>>>
>>>>> A better view server protocol is also a great idea.
>>>>>
>>>>>
>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>> Our view server is compiles functions on each view index update
>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>> others design function are works differently from views. While it's
>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>> small steps in good direction.
>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>
>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>
>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>
>>>>>> Kind regards,
>>>>>> Samuel
>>>>>
>>>>
>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

Hrm, no, I think those would remain as non_neg_integer(), but the fate of single-node is not yet determined.

B.

On 14 Jul 2014, at 19:36, Joan Touzet <wo...@apache.org> wrote:

> Yes to all 3 of your questions.
> 
> Would we mandate this format even for single-node? We could "fake it" as if it was a q=1 db, i.e.
> 
> "seq":"couchdb@foo,000-fff,12"
> 
> If you want to allow replication from 1.x hosts you'd have to be more lenient ("should" not "must").
> 
> ----- Original Message -----
> From: "Robert Samuel Newson" <rn...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Monday, July 14, 2014 10:34:21 AM
> Subject: Re: CouchDB 2.0: breaking the backward compatibility
> 
> 
> Another thought occurs; BigCouch has a different format for update_seq that is notoriously ugly.
> 
> We obviously need to encode more information for a sharded cluster (the update_seq of each shard, the range that it’s for and the node it resides on) but BigCouch also had to be compatible with CouchDB 1.x installations, so we add the sum of update sequences on the front, to trick the old replicator into working.
> 
> While it is not necessary for a human to be able to read an update_seq value (aka: you should treat it as opaque JSON), it’s often useful to unpack these for diagnostic purposes. Our use of term_to_binary confounds non-erlang libraries from doing so.
> 
> I propose we fix that in 2.0 which would require that the replicator checkpoints every N updates, and not when it sees the current update_seq exceed some delta from the update_seq of the last checkpoint.
> 
> An example readable format would be;
> 
> "seq":"couchdb@foo,000-ccc,12:couchdb@bar,d00-fff,10"
> 
> that is, a formatted string.
> 
> A few questions;
> 
> 1) Should we obscure hostnames? (we could run then forward through sha1 or even pbkdf2, akin to how .known_hosts is protected by ssh)
> 2) should we gzip encode the result?
> 3) should we base64 the result?
> 
> I think "yes" to all questions (and we would obviously have to base64 if we gzipped).
> 
> Thoughts?
> 
> B.
> 
> 
> On 13 Jul 2014, at 21:23, Paul Davis <pa...@gmail.com> wrote:
> 
>> Changing the default respones for conflicts to include all versions
>> (or no version).
>> 
>> Fix the list API (inside couchjs) so that its a pure callback like
>> everything else. Not sure if we should necessarily completely revamp
>> the whole query server protocol for 2.0. Given that its not user
>> facing I'm less inclined to think it needs to be in a major release,
>> ie we could add a new protocol in a minor release after 2.0.
>> 
>> We should rename _rev to _mvcc (or _token or _lock or anything not
>> _rev) finally.
>> 
>> Removing all metadata from document bodies has been an oft requested change.
>> 
>> Seems like there was a list of these things floating around a long time ago.
>> 
>> On Sun, Jul 13, 2014 at 3:17 PM, Robert Samuel Newson
>> <rn...@apache.org> wrote:
>>> 
>>> Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.
>>> 
>>> It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.
>>> 
>>> With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.
>>> 
>>> B.
>>> 
>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>> 
>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>> entirely from scratch.
>>>> 
>>>> -Joan
>>>> 
>>>> ----- Original Message -----
>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>> To: dev@couchdb.apache.org
>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>> 
>>>> 
>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>>>> 
>>>> A better view server protocol is also a great idea.
>>>> 
>>>> 
>>>> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>>>> 
>>>>> 
>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>> Our view server is compiles functions on each view index update
>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>> others design function are works differently from views. While it's
>>>>>> good to change and improve query server protocol completely, this task
>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>> small steps in good direction.
>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>> 
>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>> 
>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>> 
>>>>> Kind regards,
>>>>> Samuel
>>>> 
>>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Joan Touzet <wo...@apache.org>]

Yes to all 3 of your questions.

Would we mandate this format even for single-node? We could "fake it" as if it was a q=1 db, i.e.

"seq":"couchdb@foo,000-fff,12"

If you want to allow replication from 1.x hosts you'd have to be more lenient ("should" not "must").

----- Original Message -----
From: "Robert Samuel Newson" <rn...@apache.org>
To: dev@couchdb.apache.org
Sent: Monday, July 14, 2014 10:34:21 AM
Subject: Re: CouchDB 2.0: breaking the backward compatibility


Another thought occurs; BigCouch has a different format for update_seq that is notoriously ugly.

We obviously need to encode more information for a sharded cluster (the update_seq of each shard, the range that it’s for and the node it resides on) but BigCouch also had to be compatible with CouchDB 1.x installations, so we add the sum of update sequences on the front, to trick the old replicator into working.

While it is not necessary for a human to be able to read an update_seq value (aka: you should treat it as opaque JSON), it’s often useful to unpack these for diagnostic purposes. Our use of term_to_binary confounds non-erlang libraries from doing so.

I propose we fix that in 2.0 which would require that the replicator checkpoints every N updates, and not when it sees the current update_seq exceed some delta from the update_seq of the last checkpoint.

An example readable format would be;

"seq":"couchdb@foo,000-ccc,12:couchdb@bar,d00-fff,10"

that is, a formatted string.

A few questions;

1) Should we obscure hostnames? (we could run then forward through sha1 or even pbkdf2, akin to how .known_hosts is protected by ssh)
2) should we gzip encode the result?
3) should we base64 the result?

I think "yes" to all questions (and we would obviously have to base64 if we gzipped).

Thoughts?

B.


On 13 Jul 2014, at 21:23, Paul Davis <pa...@gmail.com> wrote:

> Changing the default respones for conflicts to include all versions
> (or no version).
> 
> Fix the list API (inside couchjs) so that its a pure callback like
> everything else. Not sure if we should necessarily completely revamp
> the whole query server protocol for 2.0. Given that its not user
> facing I'm less inclined to think it needs to be in a major release,
> ie we could add a new protocol in a minor release after 2.0.
> 
> We should rename _rev to _mvcc (or _token or _lock or anything not
> _rev) finally.
> 
> Removing all metadata from document bodies has been an oft requested change.
> 
> Seems like there was a list of these things floating around a long time ago.
> 
> On Sun, Jul 13, 2014 at 3:17 PM, Robert Samuel Newson
> <rn...@apache.org> wrote:
>> 
>> Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.
>> 
>> It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.
>> 
>> With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.
>> 
>> B.
>> 
>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>> 
>>> Improving the view server protocol is a great idea, but it is appropriate
>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>> entirely from scratch.
>>> 
>>> -Joan
>>> 
>>> ----- Original Message -----
>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>> To: dev@couchdb.apache.org
>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>> 
>>> 
>>> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>>> 
>>> A better view server protocol is also a great idea.
>>> 
>>> 
>>> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>>> 
>>>> 
>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>> Our view server is compiles functions on each view index update
>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>> others design function are works differently from views. While it's
>>>>> good to change and improve query server protocol completely, this task
>>>>> requires more time to be done. We should have a least plan B to do
>>>>> small steps in good direction.
>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>> 
>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>> 
>>>> I welcome people to suggest improvements/changes/ideas.
>>>> 
>>>> Kind regards,
>>>> Samuel
>>> 
>> 

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

Another thought occurs; BigCouch has a different format for update_seq that is notoriously ugly.

We obviously need to encode more information for a sharded cluster (the update_seq of each shard, the range that it’s for and the node it resides on) but BigCouch also had to be compatible with CouchDB 1.x installations, so we add the sum of update sequences on the front, to trick the old replicator into working.

While it is not necessary for a human to be able to read an update_seq value (aka: you should treat it as opaque JSON), it’s often useful to unpack these for diagnostic purposes. Our use of term_to_binary confounds non-erlang libraries from doing so.

I propose we fix that in 2.0 which would require that the replicator checkpoints every N updates, and not when it sees the current update_seq exceed some delta from the update_seq of the last checkpoint.

An example readable format would be;

"seq":"couchdb@foo,000-ccc,12:couchdb@bar,d00-fff,10"

that is, a formatted string.

A few questions;

1) Should we obscure hostnames? (we could run then forward through sha1 or even pbkdf2, akin to how .known_hosts is protected by ssh)
2) should we gzip encode the result?
3) should we base64 the result?

I think "yes" to all questions (and we would obviously have to base64 if we gzipped).

Thoughts?

B.


On 13 Jul 2014, at 21:23, Paul Davis <pa...@gmail.com> wrote:

> Changing the default respones for conflicts to include all versions
> (or no version).
> 
> Fix the list API (inside couchjs) so that its a pure callback like
> everything else. Not sure if we should necessarily completely revamp
> the whole query server protocol for 2.0. Given that its not user
> facing I'm less inclined to think it needs to be in a major release,
> ie we could add a new protocol in a minor release after 2.0.
> 
> We should rename _rev to _mvcc (or _token or _lock or anything not
> _rev) finally.
> 
> Removing all metadata from document bodies has been an oft requested change.
> 
> Seems like there was a list of these things floating around a long time ago.
> 
> On Sun, Jul 13, 2014 at 3:17 PM, Robert Samuel Newson
> <rn...@apache.org> wrote:
>> 
>> Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.
>> 
>> It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.
>> 
>> With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.
>> 
>> B.
>> 
>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>> 
>>> Improving the view server protocol is a great idea, but it is appropriate
>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>> entirely from scratch.
>>> 
>>> -Joan
>>> 
>>> ----- Original Message -----
>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>> To: dev@couchdb.apache.org
>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>> 
>>> 
>>> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>>> 
>>> A better view server protocol is also a great idea.
>>> 
>>> 
>>> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>>> 
>>>> 
>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>> Our view server is compiles functions on each view index update
>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>> others design function are works differently from views. While it's
>>>>> good to change and improve query server protocol completely, this task
>>>>> requires more time to be done. We should have a least plan B to do
>>>>> small steps in good direction.
>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>> 
>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>> 
>>>> I welcome people to suggest improvements/changes/ideas.
>>>> 
>>>> Kind regards,
>>>> Samuel
>>> 
>> 

Re: CouchDB 2.0: breaking the backward compatibility

[Paul Davis <pa...@gmail.com>]

Changing the default respones for conflicts to include all versions
(or no version).

Fix the list API (inside couchjs) so that its a pure callback like
everything else. Not sure if we should necessarily completely revamp
the whole query server protocol for 2.0. Given that its not user
facing I'm less inclined to think it needs to be in a major release,
ie we could add a new protocol in a minor release after 2.0.

We should rename _rev to _mvcc (or _token or _lock or anything not
_rev) finally.

Removing all metadata from document bodies has been an oft requested change.

Seems like there was a list of these things floating around a long time ago.

On Sun, Jul 13, 2014 at 3:17 PM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.
>
> It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.
>
> With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.
>
> B.
>
> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>
>> Improving the view server protocol is a great idea, but it is appropriate
>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>> timeframe, given 2.0 is all about merging forks, not writing new features
>> entirely from scratch.
>>
>> -Joan
>>
>> ----- Original Message -----
>> From: "Robert Samuel Newson" <rn...@apache.org>
>> To: dev@couchdb.apache.org
>> Sent: Sunday, July 13, 2014 8:52:40 AM
>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>
>>
>> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
>>
>> A better view server protocol is also a great idea.
>>
>>
>> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
>>
>>>
>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>> Our view server is compiles functions on each view index update
>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>> others design function are works differently from views. While it's
>>>> good to change and improve query server protocol completely, this task
>>>> requires more time to be done. We should have a least plan B to do
>>>> small steps in good direction.
>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>
>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>
>>> I welcome people to suggest improvements/changes/ideas.
>>>
>>> Kind regards,
>>> Samuel
>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Newson <rn...@apache.org>]

Agreed. 

Sent from my iPhone

> On 22 Jul 2014, at 07:55, "Eli Stevens (Gmail)" <wi...@gmail.com> wrote:
> 
> One quick thought:
> 
> Clustered couchdb has been a while coming, and still has a while to go
> before it's ready; I would suggest not saddling the release with
> additional changes or requirements, especially if they're the type
> that can be bikeshedded to death (stuff like API changes can take a
> while to sort out).
> 
> I'd much rather see CouchDB 4.0 sometime next year than have to wait
> until next year for 2.0.
> 
> Cheers,
> Eli
> 
> On Sat, Jul 19, 2014 at 5:23 AM, Robert Samuel Newson
> <rn...@apache.org> wrote:
>> 
>> What important tools or browsers still need text/plain for our json responses that justify the mismatch?
>> 
>> 
>>> On 19 Jul 2014, at 11:30, Jan Lehnardt <ja...@apache.org> wrote:
>>> 
>>> 
>>>> On 19 Jul 2014, at 12:27 , Robert Samuel Newson <rn...@apache.org> wrote:
>>>> 
>>>> I agree with you on the category split and, obviously, that we can make a 3.0 whenever we like, since it only indicates compatibility breaks.
>>>> 
>>>> MVCC for _security will be great since it will enable us to heal _security writes during partitions with the same logic we use for documents. Cloudant worked around (hacked around) that problem for a long time but we’ve finally done the work to add MVCC for reals, so it’ll be nice to bring that enhancement to CouchDB officially.
>>> 
>>> Yeah, I don’t see a reason not to get this in for 2.0.
>>> 
>>> 
>>>> I don’t think defaulting to conflicts=true is quite the right change. I was thinking that conflicted document would return a 300 Multiple Choices instead of a 200 OK (the response body format TBD but imagine a JSON array of each conflicting leaf revision). I agree that a change like that will break every client, but that would be the intention. A CouchDB 3.0 would be much more forthcoming about its fundamental architecture and would largely eschew the attempts to present an arbitrary "winning" revision.
>>> 
>>> Right, this was just meant as a possible compromise to get a more first-class-conflict API without making this 3-rd category change. Happy to abandon it :)
>>> 
>>> 
>>>> One further thought occurs, could we totally ditch the code that returns "text/plain" content-type? All that "are you a browser?" logic? It made sense at the time, but I feel it confuses more than it helps today.
>>> 
>>> I use that all the time :)
>>> 
>>> Best
>>> Jan
>>> --
>>> 
>>> 
>>> 
>>>> 
>>>> B.
>>>> 
>>>> 
>>>>> On 19 Jul 2014, at 10:50, Jan Lehnardt <ja...@apache.org> wrote:
>>>>> 
>>>>> 
>>>>>> On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:
>>>>>> 
>>>>>> 
>>>>>> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>>>>>> 
>>>>>> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
>>>>> 
>>>>> That is what I meant to express, with the caveat that we should be
>>>>> careful, taking a conservative stance, so we can meet in the middle.
>>>>> 
>>>>> 
>>>>>> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>>>>>> 
>>>>>> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?
>>>>> 
>>>>> 
>>>>> Most apps should continue to work on CouchDB 2.0.
>>>>> 
>>>>> Specifically, the regular document CRUD cycle should work as-is.
>>>>> Especially moving things around in the JSON usually goes further
>>>>> than the HTTP/Couch layer of most apps, as it is usually passed
>>>>> down into the rest of the app, while HTTP specifics are kept on
>>>>> the outside.
>>>>> 
>>>>> In that scenario, adding properties should be easier to do than
>>>>> removing them (e.g. _conflicts could be standard, but renaming
>>>>> _rev to _mvcc would break things more significantly), although
>>>>> Bob mentioned the replicator compatibility as a major concern,
>>>>> so we need to make sure this is doable.
>>>>> 
>>>>> My main point here is to start a discussion about how we would
>>>>> go about evolving this down the road and my suggestion was the
>>>>> separate API endpoint that we can mess with at will and gradually
>>>>> introduce until we switch at a later time when we feel confident
>>>>> that people have migrated, or a solid compatibility API is available.
>>>>> 
>>>>> I see us having three discussions:
>>>>> 
>>>>> 1. What do we want to fix/break for 2.0?
>>>>> 2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
>>>>> 3. What do we want to fix/break for later versions?
>>>>> 
>>>>> 
>>>>> From this thread, I’d handwavingly suggest these fall into category 1:
>>>>> (as per the “most apps should just continue to work”-mantra):
>>>>> 
>>>>> - timeout and heartbeat params for /_db_updates works in different way
>>>>> then the same parameters for changes feed;
>>>>> - we need to find the way to pass open_revs in POST body instead of
>>>>> tweaking max URL param;
>>>>> - we have /db/_revs_diff and /db/_revs_missing endpoints which are
>>>>> doing the same job. Well, the latter is only used for pre-1.1 CouchDB
>>>>> replicator.
>>>>> - /db/doc accepts conflicts, deleted_conflicts and revs params. In the
>>>>> same time we provides meta one which includes each of specified.
>>>>> - make eventsource feed to follow the specification format more better
>>>>> then it does now
>>>>> - MVCC for /db/_security and allow atomic changes for admins/members only
>>>>> - a variant of “Changing the default respones for conflicts to include all
>>>>> versions (or no version).” where ?conflicts=BOOL defaults to true, so we
>>>>> get an additional _conflicts: [] member on regular GETs (if there are
>>>>> conflicts), but not the conflicting versions themselves (see above note
>>>>> about additional doc members)
>>>>> - Fix the list API (inside couchjs) so that its a pure callback like
>>>>> everything else.
>>>>> - 'JSONP responses should be sent with a "application/javascript"'
>>>>> 
>>>>> 
>>>>> These fall into category 3:
>>>>> 
>>>>> - Change _rev to _mvcc or other.
>>>>> - Move document metadata elsewhere (sub-object, headers, whatever)
>>>>> - Changing the default respones for conflicts to include all versions
>>>>> (or no version).
>>>>> - more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
>>>>> - don’t pollute top level namespace (e.g. /database moves to /db/database)
>>>>> 
>>>>> This isn’t exhaustive, and we don’t yet know the answers to some of them.
>>>>> 
>>>>> As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
>>>>> 3.0 a month after 2.0, if we really want to. We are not beholden to marketing
>>>>> version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
>>>>> rather convenient :).
>>>>> 
>>>>> * * *
>>>>> 
>>>>> The view server protocol change suggested by Samuel is IMHO an internal
>>>>> change that should not break BC unless people rely on implementation details.
>>>>> 
>>>>> 
>>>>> * * *
>>>>> 
>>>>> Most apps should continue to work on CouchDB 2.0.
>>>>> 
>>>>> Jan
>>>>> --
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>>> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
>>>>> 
>>>>>> 
>>>>>> B.
>>>>>> 
>>>>>>> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>>> 
>>>>>>> I’m major -1 on substantial API changes *just* because we are having some
>>>>>>> by necessity of getting BigCouch in.
>>>>>>> 
>>>>>>> The minor improvements mentioned previously in this thread sound reasonable,
>>>>>>> but changing the main JSON format seems like a rather bad idea as it will
>>>>>>> just break all clients. While the scenario is a little different, I’d like to
>>>>>>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>>>>>>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>>>>>>> for our users, if we don’t have to).
>>>>>>> 
>>>>>>> That said, we likely want to evolve the API at some point and I think we should
>>>>>>> nail down a strategy on *how* to do that, before getting into the details of
>>>>>>> what should change.
>>>>>>> 
>>>>>>> One option, and I haven’t thought this through, would be to use separate ports
>>>>>>> for a new API endpoint that we can evolve while keeping the current one. And
>>>>>>> we can do the deprecation and switch dance some time in the future. We could
>>>>>>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>>>>>>> see, so we can learn from them). Of course there is a certain overhead in
>>>>>>> maintaining this all, and I don’t know if there are any roadblocks in the way
>>>>>>> BigCouch works today for implementing this.
>>>>>>> 
>>>>>>> Best
>>>>>>> Jan
>>>>>>> --
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>>>>>>> 
>>>>>>>> I would also love to see _rev renamed, and I think it's a good
>>>>>>>> opportunity to flip around all the meta info as well. I'm still
>>>>>>>> partial to moving the relevant metadata into the headers, and no
>>>>>>>> longer including any _* fields in the doc, but I know there are
>>>>>>>> proponents on both sides of the coin there. The most recent proposal I
>>>>>>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>>>>>>> would like to see us move all metadata into headers or into the '_'
>>>>>>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>>>>>>> it seems like an opportune time to do it.
>>>>>>>> 
>>>>>>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>>>>>>> through headers configurable. I'm not sure it's a great idea to do so,
>>>>>>>> but it's at least worth thinking about.
>>>>>>>> 
>>>>>>>> Exposing conflicts by default is another thing I'm keen on. The
>>>>>>>> question is how to make it "fail" loudly so that client libraries
>>>>>>>> don't just think it's the document body. An aggressive approach send a
>>>>>>>> list of conflict revs rather than a doc object which will break all
>>>>>>>> existing parsers and require users to deal with. Then if you want a
>>>>>>>> particular rev, you'll need to specify it in the request.
>>>>>>>> 
>>>>>>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>>>>>>> things like _all_dbs and _all_docs should be the top level endpoints
>>>>>>>> and the current info endpoints moved to _info or some such.
>>>>>>>> 
>>>>>>>> Along the lines of API cleanup is the capabilities engine. I think
>>>>>>>> this would be a great thing to land, and if done properly could be a
>>>>>>>> self defining REST endpoint showing all the things the server is
>>>>>>>> capable of and how to reach them.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> -Russell
>>>>>>>> 
>>>>>>>> 
>>>>>>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>>>>>> 
>>>>>>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>>>>>>> <rn...@apache.org> wrote:
>>>>>>>>> Great point, +1 to just making that change on master right now.
>>>>>>>>> 
>>>>>>>>> B.
>>>>>>>>> 
>>>>>>>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>>>>>>> 
>>>>>>>>>> I would like to see 'JSONP responses should be sent with a
>>>>>>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>>>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>>>>>>> change and the original issue is flying around in Jira for years.
>>>>>>>>>> 
>>>>>>>>>> Best,
>>>>>>>>>> Robert
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>>>>>>> changes.
>>>>>>>>>>> 
>>>>>>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>>>>>>> version bump.
>>>>>>>>>>> 
>>>>>>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>>>>>>> change might be.
>>>>>>>>>>> 
>>>>>>>>>>> B.
>>>>>>>>>>> 
>>>>>>>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>>>>>>> entirely from scratch.
>>>>>>>>>>>> 
>>>>>>>>>>>> -Joan
>>>>>>>>>>>> 
>>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>>>>>>> To: dev@couchdb.apache.org
>>>>>>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>>>>>> 
>>>>>>>>>>>> A better view server protocol is also a great idea.
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>>>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>>>>>>> small steps in good direction.
>>>>>>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>> Samuel
>> 

Re: CouchDB 2.0: breaking the backward compatibility

["Eli Stevens (Gmail)" <wi...@gmail.com>]

One quick thought:

Clustered couchdb has been a while coming, and still has a while to go
before it's ready; I would suggest not saddling the release with
additional changes or requirements, especially if they're the type
that can be bikeshedded to death (stuff like API changes can take a
while to sort out).

I'd much rather see CouchDB 4.0 sometime next year than have to wait
until next year for 2.0.

Cheers,
Eli

On Sat, Jul 19, 2014 at 5:23 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> What important tools or browsers still need text/plain for our json responses that justify the mismatch?
>
>
> On 19 Jul 2014, at 11:30, Jan Lehnardt <ja...@apache.org> wrote:
>
>>
>> On 19 Jul 2014, at 12:27 , Robert Samuel Newson <rn...@apache.org> wrote:
>>
>>> I agree with you on the category split and, obviously, that we can make a 3.0 whenever we like, since it only indicates compatibility breaks.
>>>
>>> MVCC for _security will be great since it will enable us to heal _security writes during partitions with the same logic we use for documents. Cloudant worked around (hacked around) that problem for a long time but we’ve finally done the work to add MVCC for reals, so it’ll be nice to bring that enhancement to CouchDB officially.
>>
>> Yeah, I don’t see a reason not to get this in for 2.0.
>>
>>
>>> I don’t think defaulting to conflicts=true is quite the right change. I was thinking that conflicted document would return a 300 Multiple Choices instead of a 200 OK (the response body format TBD but imagine a JSON array of each conflicting leaf revision). I agree that a change like that will break every client, but that would be the intention. A CouchDB 3.0 would be much more forthcoming about its fundamental architecture and would largely eschew the attempts to present an arbitrary "winning" revision.
>>
>> Right, this was just meant as a possible compromise to get a more first-class-conflict API without making this 3-rd category change. Happy to abandon it :)
>>
>>
>>> One further thought occurs, could we totally ditch the code that returns "text/plain" content-type? All that "are you a browser?" logic? It made sense at the time, but I feel it confuses more than it helps today.
>>
>> I use that all the time :)
>>
>> Best
>> Jan
>> --
>>
>>
>>
>>>
>>> B.
>>>
>>>
>>> On 19 Jul 2014, at 10:50, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>>>
>>>> On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:
>>>>
>>>>>
>>>>> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>>>>>
>>>>> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
>>>>
>>>> That is what I meant to express, with the caveat that we should be
>>>> careful, taking a conservative stance, so we can meet in the middle.
>>>>
>>>>
>>>>> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>>>>>
>>>>> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?
>>>>
>>>>
>>>> Most apps should continue to work on CouchDB 2.0.
>>>>
>>>> Specifically, the regular document CRUD cycle should work as-is.
>>>> Especially moving things around in the JSON usually goes further
>>>> than the HTTP/Couch layer of most apps, as it is usually passed
>>>> down into the rest of the app, while HTTP specifics are kept on
>>>> the outside.
>>>>
>>>> In that scenario, adding properties should be easier to do than
>>>> removing them (e.g. _conflicts could be standard, but renaming
>>>> _rev to _mvcc would break things more significantly), although
>>>> Bob mentioned the replicator compatibility as a major concern,
>>>> so we need to make sure this is doable.
>>>>
>>>> My main point here is to start a discussion about how we would
>>>> go about evolving this down the road and my suggestion was the
>>>> separate API endpoint that we can mess with at will and gradually
>>>> introduce until we switch at a later time when we feel confident
>>>> that people have migrated, or a solid compatibility API is available.
>>>>
>>>> I see us having three discussions:
>>>>
>>>> 1. What do we want to fix/break for 2.0?
>>>> 2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
>>>> 3. What do we want to fix/break for later versions?
>>>>
>>>>
>>>> From this thread, I’d handwavingly suggest these fall into category 1:
>>>> (as per the “most apps should just continue to work”-mantra):
>>>>
>>>> - timeout and heartbeat params for /_db_updates works in different way
>>>> then the same parameters for changes feed;
>>>> - we need to find the way to pass open_revs in POST body instead of
>>>> tweaking max URL param;
>>>> - we have /db/_revs_diff and /db/_revs_missing endpoints which are
>>>> doing the same job. Well, the latter is only used for pre-1.1 CouchDB
>>>> replicator.
>>>> - /db/doc accepts conflicts, deleted_conflicts and revs params. In the
>>>> same time we provides meta one which includes each of specified.
>>>> - make eventsource feed to follow the specification format more better
>>>> then it does now
>>>> - MVCC for /db/_security and allow atomic changes for admins/members only
>>>> - a variant of “Changing the default respones for conflicts to include all
>>>> versions (or no version).” where ?conflicts=BOOL defaults to true, so we
>>>> get an additional _conflicts: [] member on regular GETs (if there are
>>>> conflicts), but not the conflicting versions themselves (see above note
>>>> about additional doc members)
>>>> - Fix the list API (inside couchjs) so that its a pure callback like
>>>> everything else.
>>>> - 'JSONP responses should be sent with a "application/javascript"'
>>>>
>>>>
>>>> These fall into category 3:
>>>>
>>>> - Change _rev to _mvcc or other.
>>>> - Move document metadata elsewhere (sub-object, headers, whatever)
>>>> - Changing the default respones for conflicts to include all versions
>>>> (or no version).
>>>> - more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
>>>> - don’t pollute top level namespace (e.g. /database moves to /db/database)
>>>>
>>>> This isn’t exhaustive, and we don’t yet know the answers to some of them.
>>>>
>>>> As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
>>>> 3.0 a month after 2.0, if we really want to. We are not beholden to marketing
>>>> version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
>>>> rather convenient :).
>>>>
>>>> * * *
>>>>
>>>> The view server protocol change suggested by Samuel is IMHO an internal
>>>> change that should not break BC unless people rely on implementation details.
>>>>
>>>>
>>>> * * *
>>>>
>>>> Most apps should continue to work on CouchDB 2.0.
>>>>
>>>> Jan
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
>>>>
>>>>>
>>>>> B.
>>>>>
>>>>> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>
>>>>>> I’m major -1 on substantial API changes *just* because we are having some
>>>>>> by necessity of getting BigCouch in.
>>>>>>
>>>>>> The minor improvements mentioned previously in this thread sound reasonable,
>>>>>> but changing the main JSON format seems like a rather bad idea as it will
>>>>>> just break all clients. While the scenario is a little different, I’d like to
>>>>>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>>>>>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>>>>>> for our users, if we don’t have to).
>>>>>>
>>>>>> That said, we likely want to evolve the API at some point and I think we should
>>>>>> nail down a strategy on *how* to do that, before getting into the details of
>>>>>> what should change.
>>>>>>
>>>>>> One option, and I haven’t thought this through, would be to use separate ports
>>>>>> for a new API endpoint that we can evolve while keeping the current one. And
>>>>>> we can do the deprecation and switch dance some time in the future. We could
>>>>>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>>>>>> see, so we can learn from them). Of course there is a certain overhead in
>>>>>> maintaining this all, and I don’t know if there are any roadblocks in the way
>>>>>> BigCouch works today for implementing this.
>>>>>>
>>>>>> Best
>>>>>> Jan
>>>>>> --
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>>>>>
>>>>>>> I would also love to see _rev renamed, and I think it's a good
>>>>>>> opportunity to flip around all the meta info as well. I'm still
>>>>>>> partial to moving the relevant metadata into the headers, and no
>>>>>>> longer including any _* fields in the doc, but I know there are
>>>>>>> proponents on both sides of the coin there. The most recent proposal I
>>>>>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>>>>>> would like to see us move all metadata into headers or into the '_'
>>>>>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>>>>>> it seems like an opportune time to do it.
>>>>>>>
>>>>>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>>>>>> through headers configurable. I'm not sure it's a great idea to do so,
>>>>>>> but it's at least worth thinking about.
>>>>>>>
>>>>>>> Exposing conflicts by default is another thing I'm keen on. The
>>>>>>> question is how to make it "fail" loudly so that client libraries
>>>>>>> don't just think it's the document body. An aggressive approach send a
>>>>>>> list of conflict revs rather than a doc object which will break all
>>>>>>> existing parsers and require users to deal with. Then if you want a
>>>>>>> particular rev, you'll need to specify it in the request.
>>>>>>>
>>>>>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>>>>>> things like _all_dbs and _all_docs should be the top level endpoints
>>>>>>> and the current info endpoints moved to _info or some such.
>>>>>>>
>>>>>>> Along the lines of API cleanup is the capabilities engine. I think
>>>>>>> this would be a great thing to land, and if done properly could be a
>>>>>>> self defining REST endpoint showing all the things the server is
>>>>>>> capable of and how to reach them.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> -Russell
>>>>>>>
>>>>>>>
>>>>>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>>>>>
>>>>>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>>>>>> <rn...@apache.org> wrote:
>>>>>>>> Great point, +1 to just making that change on master right now.
>>>>>>>>
>>>>>>>> B.
>>>>>>>>
>>>>>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>>>>>
>>>>>>>>> I would like to see 'JSONP responses should be sent with a
>>>>>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>>>>>> change and the original issue is flying around in Jira for years.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Robert
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>>>>>> changes.
>>>>>>>>>>
>>>>>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>>>>>> version bump.
>>>>>>>>>>
>>>>>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>>>>>> change might be.
>>>>>>>>>>
>>>>>>>>>> B.
>>>>>>>>>>
>>>>>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>>>>>> entirely from scratch.
>>>>>>>>>>>
>>>>>>>>>>> -Joan
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>>>>>> To: dev@couchdb.apache.org
>>>>>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>>>>>
>>>>>>>>>>> A better view server protocol is also a great idea.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>>>>>> small steps in good direction.
>>>>>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>>>>>
>>>>>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>>>>>
>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>> Samuel
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

What important tools or browsers still need text/plain for our json responses that justify the mismatch?


On 19 Jul 2014, at 11:30, Jan Lehnardt <ja...@apache.org> wrote:

> 
> On 19 Jul 2014, at 12:27 , Robert Samuel Newson <rn...@apache.org> wrote:
> 
>> I agree with you on the category split and, obviously, that we can make a 3.0 whenever we like, since it only indicates compatibility breaks.
>> 
>> MVCC for _security will be great since it will enable us to heal _security writes during partitions with the same logic we use for documents. Cloudant worked around (hacked around) that problem for a long time but we’ve finally done the work to add MVCC for reals, so it’ll be nice to bring that enhancement to CouchDB officially.
> 
> Yeah, I don’t see a reason not to get this in for 2.0.
> 
> 
>> I don’t think defaulting to conflicts=true is quite the right change. I was thinking that conflicted document would return a 300 Multiple Choices instead of a 200 OK (the response body format TBD but imagine a JSON array of each conflicting leaf revision). I agree that a change like that will break every client, but that would be the intention. A CouchDB 3.0 would be much more forthcoming about its fundamental architecture and would largely eschew the attempts to present an arbitrary "winning" revision.
> 
> Right, this was just meant as a possible compromise to get a more first-class-conflict API without making this 3-rd category change. Happy to abandon it :)
> 
> 
>> One further thought occurs, could we totally ditch the code that returns "text/plain" content-type? All that "are you a browser?" logic? It made sense at the time, but I feel it confuses more than it helps today.
> 
> I use that all the time :)
> 
> Best
> Jan
> -- 
> 
> 
> 
>> 
>> B.
>> 
>> 
>> On 19 Jul 2014, at 10:50, Jan Lehnardt <ja...@apache.org> wrote:
>> 
>>> 
>>> On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:
>>> 
>>>> 
>>>> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>>>> 
>>>> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
>>> 
>>> That is what I meant to express, with the caveat that we should be
>>> careful, taking a conservative stance, so we can meet in the middle.
>>> 
>>> 
>>>> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>>>> 
>>>> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?
>>> 
>>> 
>>> Most apps should continue to work on CouchDB 2.0.
>>> 
>>> Specifically, the regular document CRUD cycle should work as-is.
>>> Especially moving things around in the JSON usually goes further
>>> than the HTTP/Couch layer of most apps, as it is usually passed
>>> down into the rest of the app, while HTTP specifics are kept on
>>> the outside.
>>> 
>>> In that scenario, adding properties should be easier to do than
>>> removing them (e.g. _conflicts could be standard, but renaming
>>> _rev to _mvcc would break things more significantly), although
>>> Bob mentioned the replicator compatibility as a major concern,
>>> so we need to make sure this is doable.
>>> 
>>> My main point here is to start a discussion about how we would
>>> go about evolving this down the road and my suggestion was the
>>> separate API endpoint that we can mess with at will and gradually
>>> introduce until we switch at a later time when we feel confident
>>> that people have migrated, or a solid compatibility API is available.
>>> 
>>> I see us having three discussions:
>>> 
>>> 1. What do we want to fix/break for 2.0?
>>> 2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
>>> 3. What do we want to fix/break for later versions?
>>> 
>>> 
>>> From this thread, I’d handwavingly suggest these fall into category 1:
>>> (as per the “most apps should just continue to work”-mantra):
>>> 
>>> - timeout and heartbeat params for /_db_updates works in different way
>>> then the same parameters for changes feed;
>>> - we need to find the way to pass open_revs in POST body instead of
>>> tweaking max URL param;
>>> - we have /db/_revs_diff and /db/_revs_missing endpoints which are
>>> doing the same job. Well, the latter is only used for pre-1.1 CouchDB
>>> replicator.
>>> - /db/doc accepts conflicts, deleted_conflicts and revs params. In the
>>> same time we provides meta one which includes each of specified.
>>> - make eventsource feed to follow the specification format more better
>>> then it does now
>>> - MVCC for /db/_security and allow atomic changes for admins/members only
>>> - a variant of “Changing the default respones for conflicts to include all
>>> versions (or no version).” where ?conflicts=BOOL defaults to true, so we
>>> get an additional _conflicts: [] member on regular GETs (if there are
>>> conflicts), but not the conflicting versions themselves (see above note
>>> about additional doc members)
>>> - Fix the list API (inside couchjs) so that its a pure callback like
>>> everything else.
>>> - 'JSONP responses should be sent with a "application/javascript"'
>>> 
>>> 
>>> These fall into category 3:
>>> 
>>> - Change _rev to _mvcc or other.
>>> - Move document metadata elsewhere (sub-object, headers, whatever)
>>> - Changing the default respones for conflicts to include all versions
>>> (or no version).
>>> - more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
>>> - don’t pollute top level namespace (e.g. /database moves to /db/database)
>>> 
>>> This isn’t exhaustive, and we don’t yet know the answers to some of them.
>>> 
>>> As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
>>> 3.0 a month after 2.0, if we really want to. We are not beholden to marketing
>>> version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
>>> rather convenient :).
>>> 
>>> * * *
>>> 
>>> The view server protocol change suggested by Samuel is IMHO an internal
>>> change that should not break BC unless people rely on implementation details.
>>> 
>>> 
>>> * * *
>>> 
>>> Most apps should continue to work on CouchDB 2.0.
>>> 
>>> Jan
>>> -- 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>>> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
>>> 
>>>> 
>>>> B.
>>>> 
>>>> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>>>> 
>>>>> I’m major -1 on substantial API changes *just* because we are having some
>>>>> by necessity of getting BigCouch in.
>>>>> 
>>>>> The minor improvements mentioned previously in this thread sound reasonable,
>>>>> but changing the main JSON format seems like a rather bad idea as it will
>>>>> just break all clients. While the scenario is a little different, I’d like to
>>>>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>>>>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>>>>> for our users, if we don’t have to).
>>>>> 
>>>>> That said, we likely want to evolve the API at some point and I think we should
>>>>> nail down a strategy on *how* to do that, before getting into the details of
>>>>> what should change.
>>>>> 
>>>>> One option, and I haven’t thought this through, would be to use separate ports
>>>>> for a new API endpoint that we can evolve while keeping the current one. And
>>>>> we can do the deprecation and switch dance some time in the future. We could
>>>>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>>>>> see, so we can learn from them). Of course there is a certain overhead in
>>>>> maintaining this all, and I don’t know if there are any roadblocks in the way
>>>>> BigCouch works today for implementing this.
>>>>> 
>>>>> Best
>>>>> Jan
>>>>> --
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>>>> 
>>>>>> I would also love to see _rev renamed, and I think it's a good
>>>>>> opportunity to flip around all the meta info as well. I'm still
>>>>>> partial to moving the relevant metadata into the headers, and no
>>>>>> longer including any _* fields in the doc, but I know there are
>>>>>> proponents on both sides of the coin there. The most recent proposal I
>>>>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>>>>> would like to see us move all metadata into headers or into the '_'
>>>>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>>>>> it seems like an opportune time to do it.
>>>>>> 
>>>>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>>>>> through headers configurable. I'm not sure it's a great idea to do so,
>>>>>> but it's at least worth thinking about.
>>>>>> 
>>>>>> Exposing conflicts by default is another thing I'm keen on. The
>>>>>> question is how to make it "fail" loudly so that client libraries
>>>>>> don't just think it's the document body. An aggressive approach send a
>>>>>> list of conflict revs rather than a doc object which will break all
>>>>>> existing parsers and require users to deal with. Then if you want a
>>>>>> particular rev, you'll need to specify it in the request.
>>>>>> 
>>>>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>>>>> things like _all_dbs and _all_docs should be the top level endpoints
>>>>>> and the current info endpoints moved to _info or some such.
>>>>>> 
>>>>>> Along the lines of API cleanup is the capabilities engine. I think
>>>>>> this would be a great thing to land, and if done properly could be a
>>>>>> self defining REST endpoint showing all the things the server is
>>>>>> capable of and how to reach them.
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> -Russell
>>>>>> 
>>>>>> 
>>>>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>>>> 
>>>>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>>>>> <rn...@apache.org> wrote:
>>>>>>> Great point, +1 to just making that change on master right now.
>>>>>>> 
>>>>>>> B.
>>>>>>> 
>>>>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>>>> 
>>>>>>>> I would like to see 'JSONP responses should be sent with a
>>>>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>>>>> change and the original issue is flying around in Jira for years.
>>>>>>>> 
>>>>>>>> Best,
>>>>>>>> Robert
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>>>>> changes.
>>>>>>>>> 
>>>>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>>>>> version bump.
>>>>>>>>> 
>>>>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>>>>> change might be.
>>>>>>>>> 
>>>>>>>>> B.
>>>>>>>>> 
>>>>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>>>> 
>>>>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>>>>> entirely from scratch.
>>>>>>>>>> 
>>>>>>>>>> -Joan
>>>>>>>>>> 
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>>>>> To: dev@couchdb.apache.org
>>>>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>>>> 
>>>>>>>>>> A better view server protocol is also a great idea.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>>>>> small steps in good direction.
>>>>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>>>> 
>>>>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>>>> 
>>>>>>>>>>> Kind regards,
>>>>>>>>>>> Samuel
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Jan Lehnardt <ja...@apache.org>]

On 19 Jul 2014, at 12:27 , Robert Samuel Newson <rn...@apache.org> wrote:

> I agree with you on the category split and, obviously, that we can make a 3.0 whenever we like, since it only indicates compatibility breaks.
> 
> MVCC for _security will be great since it will enable us to heal _security writes during partitions with the same logic we use for documents. Cloudant worked around (hacked around) that problem for a long time but we’ve finally done the work to add MVCC for reals, so it’ll be nice to bring that enhancement to CouchDB officially.

Yeah, I don’t see a reason not to get this in for 2.0.


> I don’t think defaulting to conflicts=true is quite the right change. I was thinking that conflicted document would return a 300 Multiple Choices instead of a 200 OK (the response body format TBD but imagine a JSON array of each conflicting leaf revision). I agree that a change like that will break every client, but that would be the intention. A CouchDB 3.0 would be much more forthcoming about its fundamental architecture and would largely eschew the attempts to present an arbitrary "winning" revision.

Right, this was just meant as a possible compromise to get a more first-class-conflict API without making this 3-rd category change. Happy to abandon it :)


> One further thought occurs, could we totally ditch the code that returns "text/plain" content-type? All that "are you a browser?" logic? It made sense at the time, but I feel it confuses more than it helps today.

I use that all the time :)

Best
Jan
-- 



> 
> B.
> 
> 
> On 19 Jul 2014, at 10:50, Jan Lehnardt <ja...@apache.org> wrote:
> 
>> 
>> On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:
>> 
>>> 
>>> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>>> 
>>> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
>> 
>> That is what I meant to express, with the caveat that we should be
>> careful, taking a conservative stance, so we can meet in the middle.
>> 
>> 
>>> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>>> 
>>> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?
>> 
>> 
>> Most apps should continue to work on CouchDB 2.0.
>> 
>> Specifically, the regular document CRUD cycle should work as-is.
>> Especially moving things around in the JSON usually goes further
>> than the HTTP/Couch layer of most apps, as it is usually passed
>> down into the rest of the app, while HTTP specifics are kept on
>> the outside.
>> 
>> In that scenario, adding properties should be easier to do than
>> removing them (e.g. _conflicts could be standard, but renaming
>> _rev to _mvcc would break things more significantly), although
>> Bob mentioned the replicator compatibility as a major concern,
>> so we need to make sure this is doable.
>> 
>> My main point here is to start a discussion about how we would
>> go about evolving this down the road and my suggestion was the
>> separate API endpoint that we can mess with at will and gradually
>> introduce until we switch at a later time when we feel confident
>> that people have migrated, or a solid compatibility API is available.
>> 
>> I see us having three discussions:
>> 
>> 1. What do we want to fix/break for 2.0?
>> 2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
>> 3. What do we want to fix/break for later versions?
>> 
>> 
>> From this thread, I’d handwavingly suggest these fall into category 1:
>> (as per the “most apps should just continue to work”-mantra):
>> 
>> - timeout and heartbeat params for /_db_updates works in different way
>> then the same parameters for changes feed;
>> - we need to find the way to pass open_revs in POST body instead of
>> tweaking max URL param;
>> - we have /db/_revs_diff and /db/_revs_missing endpoints which are
>> doing the same job. Well, the latter is only used for pre-1.1 CouchDB
>> replicator.
>> - /db/doc accepts conflicts, deleted_conflicts and revs params. In the
>> same time we provides meta one which includes each of specified.
>> - make eventsource feed to follow the specification format more better
>> then it does now
>> - MVCC for /db/_security and allow atomic changes for admins/members only
>> - a variant of “Changing the default respones for conflicts to include all
>> versions (or no version).” where ?conflicts=BOOL defaults to true, so we
>> get an additional _conflicts: [] member on regular GETs (if there are
>> conflicts), but not the conflicting versions themselves (see above note
>> about additional doc members)
>> - Fix the list API (inside couchjs) so that its a pure callback like
>> everything else.
>> - 'JSONP responses should be sent with a "application/javascript"'
>> 
>> 
>> These fall into category 3:
>> 
>> - Change _rev to _mvcc or other.
>> - Move document metadata elsewhere (sub-object, headers, whatever)
>> - Changing the default respones for conflicts to include all versions
>> (or no version).
>> - more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
>> - don’t pollute top level namespace (e.g. /database moves to /db/database)
>> 
>> This isn’t exhaustive, and we don’t yet know the answers to some of them.
>> 
>> As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
>> 3.0 a month after 2.0, if we really want to. We are not beholden to marketing
>> version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
>> rather convenient :).
>> 
>> * * *
>> 
>> The view server protocol change suggested by Samuel is IMHO an internal
>> change that should not break BC unless people rely on implementation details.
>> 
>> 
>> * * *
>> 
>> Most apps should continue to work on CouchDB 2.0.
>> 
>> Jan
>> -- 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
>> 
>>> 
>>> B.
>>> 
>>> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>>> 
>>>> I’m major -1 on substantial API changes *just* because we are having some
>>>> by necessity of getting BigCouch in.
>>>> 
>>>> The minor improvements mentioned previously in this thread sound reasonable,
>>>> but changing the main JSON format seems like a rather bad idea as it will
>>>> just break all clients. While the scenario is a little different, I’d like to
>>>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>>>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>>>> for our users, if we don’t have to).
>>>> 
>>>> That said, we likely want to evolve the API at some point and I think we should
>>>> nail down a strategy on *how* to do that, before getting into the details of
>>>> what should change.
>>>> 
>>>> One option, and I haven’t thought this through, would be to use separate ports
>>>> for a new API endpoint that we can evolve while keeping the current one. And
>>>> we can do the deprecation and switch dance some time in the future. We could
>>>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>>>> see, so we can learn from them). Of course there is a certain overhead in
>>>> maintaining this all, and I don’t know if there are any roadblocks in the way
>>>> BigCouch works today for implementing this.
>>>> 
>>>> Best
>>>> Jan
>>>> --
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>>> 
>>>>> I would also love to see _rev renamed, and I think it's a good
>>>>> opportunity to flip around all the meta info as well. I'm still
>>>>> partial to moving the relevant metadata into the headers, and no
>>>>> longer including any _* fields in the doc, but I know there are
>>>>> proponents on both sides of the coin there. The most recent proposal I
>>>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>>>> would like to see us move all metadata into headers or into the '_'
>>>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>>>> it seems like an opportune time to do it.
>>>>> 
>>>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>>>> through headers configurable. I'm not sure it's a great idea to do so,
>>>>> but it's at least worth thinking about.
>>>>> 
>>>>> Exposing conflicts by default is another thing I'm keen on. The
>>>>> question is how to make it "fail" loudly so that client libraries
>>>>> don't just think it's the document body. An aggressive approach send a
>>>>> list of conflict revs rather than a doc object which will break all
>>>>> existing parsers and require users to deal with. Then if you want a
>>>>> particular rev, you'll need to specify it in the request.
>>>>> 
>>>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>>>> things like _all_dbs and _all_docs should be the top level endpoints
>>>>> and the current info endpoints moved to _info or some such.
>>>>> 
>>>>> Along the lines of API cleanup is the capabilities engine. I think
>>>>> this would be a great thing to land, and if done properly could be a
>>>>> self defining REST endpoint showing all the things the server is
>>>>> capable of and how to reach them.
>>>>> 
>>>>> 
>>>>> 
>>>>> -Russell
>>>>> 
>>>>> 
>>>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>>> 
>>>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>>>> <rn...@apache.org> wrote:
>>>>>> Great point, +1 to just making that change on master right now.
>>>>>> 
>>>>>> B.
>>>>>> 
>>>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>>> 
>>>>>>> I would like to see 'JSONP responses should be sent with a
>>>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>>>> change and the original issue is flying around in Jira for years.
>>>>>>> 
>>>>>>> Best,
>>>>>>> Robert
>>>>>>> 
>>>>>>> 
>>>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>>> 
>>>>>>>> 
>>>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>>>> changes.
>>>>>>>> 
>>>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>>>> version bump.
>>>>>>>> 
>>>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>>>> change might be.
>>>>>>>> 
>>>>>>>> B.
>>>>>>>> 
>>>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>>> 
>>>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>>>> entirely from scratch.
>>>>>>>>> 
>>>>>>>>> -Joan
>>>>>>>>> 
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>>>> To: dev@couchdb.apache.org
>>>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>>> 
>>>>>>>>> A better view server protocol is also a great idea.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>>>> small steps in good direction.
>>>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>>> 
>>>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>>> 
>>>>>>>>>> Kind regards,
>>>>>>>>>> Samuel
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

I agree with you on the category split and, obviously, that we can make a 3.0 whenever we like, since it only indicates compatibility breaks.

MVCC for _security will be great since it will enable us to heal _security writes during partitions with the same logic we use for documents. Cloudant worked around (hacked around) that problem for a long time but we’ve finally done the work to add MVCC for reals, so it’ll be nice to bring that enhancement to CouchDB officially.

I don’t think defaulting to conflicts=true is quite the right change. I was thinking that conflicted document would return a 300 Multiple Choices instead of a 200 OK (the response body format TBD but imagine a JSON array of each conflicting leaf revision). I agree that a change like that will break every client, but that would be the intention. A CouchDB 3.0 would be much more forthcoming about its fundamental architecture and would largely eschew the attempts to present an arbitrary "winning" revision.

One further thought occurs, could we totally ditch the code that returns "text/plain" content-type? All that "are you a browser?" logic? It made sense at the time, but I feel it confuses more than it helps today.

B.


On 19 Jul 2014, at 10:50, Jan Lehnardt <ja...@apache.org> wrote:

> 
> On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:
> 
>> 
>> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>> 
>> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
> 
> That is what I meant to express, with the caveat that we should be
> careful, taking a conservative stance, so we can meet in the middle.
> 
> 
>> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>> 
>> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?
> 
> 
> Most apps should continue to work on CouchDB 2.0.
> 
> Specifically, the regular document CRUD cycle should work as-is.
> Especially moving things around in the JSON usually goes further
> than the HTTP/Couch layer of most apps, as it is usually passed
> down into the rest of the app, while HTTP specifics are kept on
> the outside.
> 
> In that scenario, adding properties should be easier to do than
> removing them (e.g. _conflicts could be standard, but renaming
> _rev to _mvcc would break things more significantly), although
> Bob mentioned the replicator compatibility as a major concern,
> so we need to make sure this is doable.
> 
> My main point here is to start a discussion about how we would
> go about evolving this down the road and my suggestion was the
> separate API endpoint that we can mess with at will and gradually
> introduce until we switch at a later time when we feel confident
> that people have migrated, or a solid compatibility API is available.
> 
> I see us having three discussions:
> 
> 1. What do we want to fix/break for 2.0?
> 2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
> 3. What do we want to fix/break for later versions?
> 
> 
> From this thread, I’d handwavingly suggest these fall into category 1:
> (as per the “most apps should just continue to work”-mantra):
> 
> - timeout and heartbeat params for /_db_updates works in different way
> then the same parameters for changes feed;
> - we need to find the way to pass open_revs in POST body instead of
> tweaking max URL param;
> - we have /db/_revs_diff and /db/_revs_missing endpoints which are
> doing the same job. Well, the latter is only used for pre-1.1 CouchDB
> replicator.
> - /db/doc accepts conflicts, deleted_conflicts and revs params. In the
> same time we provides meta one which includes each of specified.
> - make eventsource feed to follow the specification format more better
> then it does now
> - MVCC for /db/_security and allow atomic changes for admins/members only
> - a variant of “Changing the default respones for conflicts to include all
> versions (or no version).” where ?conflicts=BOOL defaults to true, so we
> get an additional _conflicts: [] member on regular GETs (if there are
> conflicts), but not the conflicting versions themselves (see above note
> about additional doc members)
> - Fix the list API (inside couchjs) so that its a pure callback like
> everything else.
> - 'JSONP responses should be sent with a "application/javascript"'
> 
> 
> These fall into category 3:
> 
> - Change _rev to _mvcc or other.
> - Move document metadata elsewhere (sub-object, headers, whatever)
> - Changing the default respones for conflicts to include all versions
> (or no version).
> - more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
> - don’t pollute top level namespace (e.g. /database moves to /db/database)
> 
> This isn’t exhaustive, and we don’t yet know the answers to some of them.
> 
> As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
> 3.0 a month after 2.0, if we really want to. We are not beholden to marketing
> version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
> rather convenient :).
> 
> * * *
> 
> The view server protocol change suggested by Samuel is IMHO an internal
> change that should not break BC unless people rely on implementation details.
> 
> 
> * * *
> 
> Most apps should continue to work on CouchDB 2.0.
> 
> Jan
> -- 
> 
> 
> 
> 
> 
> 
> 
> 
>> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
> 
>> 
>> B.
>> 
>> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>> 
>>> I’m major -1 on substantial API changes *just* because we are having some
>>> by necessity of getting BigCouch in.
>>> 
>>> The minor improvements mentioned previously in this thread sound reasonable,
>>> but changing the main JSON format seems like a rather bad idea as it will
>>> just break all clients. While the scenario is a little different, I’d like to
>>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>>> for our users, if we don’t have to).
>>> 
>>> That said, we likely want to evolve the API at some point and I think we should
>>> nail down a strategy on *how* to do that, before getting into the details of
>>> what should change.
>>> 
>>> One option, and I haven’t thought this through, would be to use separate ports
>>> for a new API endpoint that we can evolve while keeping the current one. And
>>> we can do the deprecation and switch dance some time in the future. We could
>>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>>> see, so we can learn from them). Of course there is a certain overhead in
>>> maintaining this all, and I don’t know if there are any roadblocks in the way
>>> BigCouch works today for implementing this.
>>> 
>>> Best
>>> Jan
>>> --
>>> 
>>> 
>>> 
>>> 
>>> 
>>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>> 
>>>> I would also love to see _rev renamed, and I think it's a good
>>>> opportunity to flip around all the meta info as well. I'm still
>>>> partial to moving the relevant metadata into the headers, and no
>>>> longer including any _* fields in the doc, but I know there are
>>>> proponents on both sides of the coin there. The most recent proposal I
>>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>>> would like to see us move all metadata into headers or into the '_'
>>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>>> it seems like an opportune time to do it.
>>>> 
>>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>>> through headers configurable. I'm not sure it's a great idea to do so,
>>>> but it's at least worth thinking about.
>>>> 
>>>> Exposing conflicts by default is another thing I'm keen on. The
>>>> question is how to make it "fail" loudly so that client libraries
>>>> don't just think it's the document body. An aggressive approach send a
>>>> list of conflict revs rather than a doc object which will break all
>>>> existing parsers and require users to deal with. Then if you want a
>>>> particular rev, you'll need to specify it in the request.
>>>> 
>>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>>> things like _all_dbs and _all_docs should be the top level endpoints
>>>> and the current info endpoints moved to _info or some such.
>>>> 
>>>> Along the lines of API cleanup is the capabilities engine. I think
>>>> this would be a great thing to land, and if done properly could be a
>>>> self defining REST endpoint showing all the things the server is
>>>> capable of and how to reach them.
>>>> 
>>>> 
>>>> 
>>>> -Russell
>>>> 
>>>> 
>>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>> 
>>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>>> <rn...@apache.org> wrote:
>>>>> Great point, +1 to just making that change on master right now.
>>>>> 
>>>>> B.
>>>>> 
>>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>> 
>>>>>> I would like to see 'JSONP responses should be sent with a
>>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>>> change and the original issue is flying around in Jira for years.
>>>>>> 
>>>>>> Best,
>>>>>> Robert
>>>>>> 
>>>>>> 
>>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>> 
>>>>>>> 
>>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>>> changes.
>>>>>>> 
>>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>>> version bump.
>>>>>>> 
>>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>>> change might be.
>>>>>>> 
>>>>>>> B.
>>>>>>> 
>>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>> 
>>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>>> entirely from scratch.
>>>>>>>> 
>>>>>>>> -Joan
>>>>>>>> 
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>>> To: dev@couchdb.apache.org
>>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>> 
>>>>>>>> A better view server protocol is also a great idea.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>>> small steps in good direction.
>>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>> 
>>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>> 
>>>>>>>>> Kind regards,
>>>>>>>>> Samuel
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Jan Lehnardt <ja...@apache.org>]

On 19 Jul 2014, at 10:38 , Robert Samuel Newson <rn...@apache.org> wrote:

> 
> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
> 
> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.

That is what I meant to express, with the caveat that we should be
careful, taking a conservative stance, so we can meet in the middle.


> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
> 
> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope?


Most apps should continue to work on CouchDB 2.0.

Specifically, the regular document CRUD cycle should work as-is.
Especially moving things around in the JSON usually goes further
than the HTTP/Couch layer of most apps, as it is usually passed
down into the rest of the app, while HTTP specifics are kept on
the outside.

In that scenario, adding properties should be easier to do than
removing them (e.g. _conflicts could be standard, but renaming
_rev to _mvcc would break things more significantly), although
Bob mentioned the replicator compatibility as a major concern,
so we need to make sure this is doable.

My main point here is to start a discussion about how we would
go about evolving this down the road and my suggestion was the
separate API endpoint that we can mess with at will and gradually
introduce until we switch at a later time when we feel confident
that people have migrated, or a solid compatibility API is available.

I see us having three discussions:

1. What do we want to fix/break for 2.0?
2. How do we introduce fixes/breaks that we aren’t comfortable doing for 2.0?
3. What do we want to fix/break for later versions?


From this thread, I’d handwavingly suggest these fall into category 1:
(as per the “most apps should just continue to work”-mantra):

- timeout and heartbeat params for /_db_updates works in different way
then the same parameters for changes feed;
- we need to find the way to pass open_revs in POST body instead of
tweaking max URL param;
- we have /db/_revs_diff and /db/_revs_missing endpoints which are
doing the same job. Well, the latter is only used for pre-1.1 CouchDB
replicator.
- /db/doc accepts conflicts, deleted_conflicts and revs params. In the
same time we provides meta one which includes each of specified.
- make eventsource feed to follow the specification format more better
then it does now
- MVCC for /db/_security and allow atomic changes for admins/members only
- a variant of “Changing the default respones for conflicts to include all
versions (or no version).” where ?conflicts=BOOL defaults to true, so we
get an additional _conflicts: [] member on regular GETs (if there are
conflicts), but not the conflicting versions themselves (see above note
about additional doc members)
- Fix the list API (inside couchjs) so that its a pure callback like
everything else.
- 'JSONP responses should be sent with a "application/javascript"'


These fall into category 3:

- Change _rev to _mvcc or other.
- Move document metadata elsewhere (sub-object, headers, whatever)
- Changing the default respones for conflicts to include all versions
(or no version).
- more RESTy API (move /_all_docs to /, db info to _info etc), self-defining REST API
- don’t pollute top level namespace (e.g. /database moves to /db/database)

This isn’t exhaustive, and we don’t yet know the answers to some of them.

As a repeat: with our new understanding of SemVer, we are free to ship CouchDB
3.0 a month after 2.0, if we really want to. We are not beholden to marketing
version numbers after 2.0 (strictly, we aren’t for 2.0 either, but it is
rather convenient :).

* * *

The view server protocol change suggested by Samuel is IMHO an internal
change that should not break BC unless people rely on implementation details.


* * *

Most apps should continue to work on CouchDB 2.0.

Jan
-- 








> Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.

> 
> B.
> 
> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
> 
>> I’m major -1 on substantial API changes *just* because we are having some
>> by necessity of getting BigCouch in.
>> 
>> The minor improvements mentioned previously in this thread sound reasonable,
>> but changing the main JSON format seems like a rather bad idea as it will
>> just break all clients. While the scenario is a little different, I’d like to
>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>> for our users, if we don’t have to).
>> 
>> That said, we likely want to evolve the API at some point and I think we should
>> nail down a strategy on *how* to do that, before getting into the details of
>> what should change.
>> 
>> One option, and I haven’t thought this through, would be to use separate ports
>> for a new API endpoint that we can evolve while keeping the current one. And
>> we can do the deprecation and switch dance some time in the future. We could
>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>> see, so we can learn from them). Of course there is a certain overhead in
>> maintaining this all, and I don’t know if there are any roadblocks in the way
>> BigCouch works today for implementing this.
>> 
>> Best
>> Jan
>> --
>> 
>> 
>> 
>> 
>> 
>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>> 
>>> I would also love to see _rev renamed, and I think it's a good
>>> opportunity to flip around all the meta info as well. I'm still
>>> partial to moving the relevant metadata into the headers, and no
>>> longer including any _* fields in the doc, but I know there are
>>> proponents on both sides of the coin there. The most recent proposal I
>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>> would like to see us move all metadata into headers or into the '_'
>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>> it seems like an opportune time to do it.
>>> 
>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>> through headers configurable. I'm not sure it's a great idea to do so,
>>> but it's at least worth thinking about.
>>> 
>>> Exposing conflicts by default is another thing I'm keen on. The
>>> question is how to make it "fail" loudly so that client libraries
>>> don't just think it's the document body. An aggressive approach send a
>>> list of conflict revs rather than a doc object which will break all
>>> existing parsers and require users to deal with. Then if you want a
>>> particular rev, you'll need to specify it in the request.
>>> 
>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>> things like _all_dbs and _all_docs should be the top level endpoints
>>> and the current info endpoints moved to _info or some such.
>>> 
>>> Along the lines of API cleanup is the capabilities engine. I think
>>> this would be a great thing to land, and if done properly could be a
>>> self defining REST endpoint showing all the things the server is
>>> capable of and how to reach them.
>>> 
>>> 
>>> 
>>> -Russell
>>> 
>>> 
>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>> 
>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>> <rn...@apache.org> wrote:
>>>> Great point, +1 to just making that change on master right now.
>>>> 
>>>> B.
>>>> 
>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>> 
>>>>> I would like to see 'JSONP responses should be sent with a
>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>> change and the original issue is flying around in Jira for years.
>>>>> 
>>>>> Best,
>>>>> Robert
>>>>> 
>>>>> 
>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>> 
>>>>>> 
>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>> changes.
>>>>>> 
>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>> version bump.
>>>>>> 
>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>> change might be.
>>>>>> 
>>>>>> B.
>>>>>> 
>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>> 
>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>> entirely from scratch.
>>>>>>> 
>>>>>>> -Joan
>>>>>>> 
>>>>>>> ----- Original Message -----
>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>> To: dev@couchdb.apache.org
>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>> 
>>>>>>> 
>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>> 
>>>>>>> A better view server protocol is also a great idea.
>>>>>>> 
>>>>>>> 
>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>> small steps in good direction.
>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>> 
>>>>>>>> 
>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>> 
>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>> 
>>>>>>>> Kind regards,
>>>>>>>> Samuel
>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Benoit Chesneau <bc...@gmail.com>]

On Sat, Jul 19, 2014 at 10:38 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
>
> I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.
>
> Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.
>
> I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.
>
> Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope? Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.
>
> B.

If we don't do these changes on 2.0 then will it needs to be done on a
3.0? I guess so. I think we should take the opportunity of this
release to make the changes. Possibly with a compatibility with the
old API but it may be hard for some stuff.

- benoit


>
> On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:
>
>> I’m major -1 on substantial API changes *just* because we are having some
>> by necessity of getting BigCouch in.
>>
>> The minor improvements mentioned previously in this thread sound reasonable,
>> but changing the main JSON format seems like a rather bad idea as it will
>> just break all clients. While the scenario is a little different, I’d like to
>> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
>> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
>> for our users, if we don’t have to).
>>
>> That said, we likely want to evolve the API at some point and I think we should
>> nail down a strategy on *how* to do that, before getting into the details of
>> what should change.
>>
>> One option, and I haven’t thought this through, would be to use separate ports
>> for a new API endpoint that we can evolve while keeping the current one. And
>> we can do the deprecation and switch dance some time in the future. We could
>> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
>> see, so we can learn from them). Of course there is a certain overhead in
>> maintaining this all, and I don’t know if there are any roadblocks in the way
>> BigCouch works today for implementing this.
>>
>> Best
>> Jan
>> --
>>
>>
>>
>>
>>
>> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
>>
>>> I would also love to see _rev renamed, and I think it's a good
>>> opportunity to flip around all the meta info as well. I'm still
>>> partial to moving the relevant metadata into the headers, and no
>>> longer including any _* fields in the doc, but I know there are
>>> proponents on both sides of the coin there. The most recent proposal I
>>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>>> would like to see us move all metadata into headers or into the '_'
>>> field, and rename 'rev'. There's a lot of code overlap for the two so
>>> it seems like an opportune time to do it.
>>>
>>> I wonder if it's reasonable to make the use of a '_' field or exposed
>>> through headers configurable. I'm not sure it's a great idea to do so,
>>> but it's at least worth thinking about.
>>>
>>> Exposing conflicts by default is another thing I'm keen on. The
>>> question is how to make it "fail" loudly so that client libraries
>>> don't just think it's the document body. An aggressive approach send a
>>> list of conflict revs rather than a doc object which will break all
>>> existing parsers and require users to deal with. Then if you want a
>>> particular rev, you'll need to specify it in the request.
>>>
>>> We could also cleanup the API endpoints to make them more RESTful. IMO
>>> things like _all_dbs and _all_docs should be the top level endpoints
>>> and the current info endpoints moved to _info or some such.
>>>
>>> Along the lines of API cleanup is the capabilities engine. I think
>>> this would be a great thing to land, and if done properly could be a
>>> self defining REST endpoint showing all the things the server is
>>> capable of and how to reach them.
>>>
>>>
>>>
>>> -Russell
>>>
>>>
>>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>>>
>>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>>> <rn...@apache.org> wrote:
>>>> Great point, +1 to just making that change on master right now.
>>>>
>>>> B.
>>>>
>>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>>>
>>>>> I would like to see 'JSONP responses should be sent with a
>>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>>> change and the original issue is flying around in Jira for years.
>>>>>
>>>>> Best,
>>>>> Robert
>>>>>
>>>>>
>>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>>>
>>>>>>
>>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>>> changes.
>>>>>>
>>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>>> version bump.
>>>>>>
>>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>>> change might be.
>>>>>>
>>>>>> B.
>>>>>>
>>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>>>
>>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>>> entirely from scratch.
>>>>>>>
>>>>>>> -Joan
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>>> To: dev@couchdb.apache.org
>>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>>>
>>>>>>>
>>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>>> so very recently, so I will be pulling that work over soon).
>>>>>>>
>>>>>>> A better view server protocol is also a great idea.
>>>>>>>
>>>>>>>
>>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>>> others design function are works differently from views. While it's
>>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>>> small steps in good direction.
>>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>>>
>>>>>>>>
>>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>>>
>>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Samuel
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

I think this is backward. We are not proposing API changes "just because" BigCouch happens to make them.

Given that we have to bump the major version number, we are afforded an opportunity to improve our API in significant ways for the first time since 1.0. We all know there are warts to be fixed. The question is what to fix with 2.0, seeing as we’re making one.

I would also hate to see low adoption of CouchDB 2.0 if we change too much, because I’ve spent quite a bit of time trying to make it happen.

Is there nothing, besides what comes with the BigCouch merge, that we wouldn’t want to change for 2.0? At least the ability to add metadata without breaking 2.0 compliant clients and libraries, I hope? Adding a new top-level _-prefixed field in couchdb causes the replicator to crash hard, this is unacceptable brittle imo.

B.

On 18 Jul 2014, at 21:15, Jan Lehnardt <ja...@apache.org> wrote:

> I’m major -1 on substantial API changes *just* because we are having some
> by necessity of getting BigCouch in.
> 
> The minor improvements mentioned previously in this thread sound reasonable,
> but changing the main JSON format seems like a rather bad idea as it will
> just break all clients. While the scenario is a little different, I’d like to
> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
> for our users, if we don’t have to).
> 
> That said, we likely want to evolve the API at some point and I think we should
> nail down a strategy on *how* to do that, before getting into the details of
> what should change.
> 
> One option, and I haven’t thought this through, would be to use separate ports
> for a new API endpoint that we can evolve while keeping the current one. And
> we can do the deprecation and switch dance some time in the future. We could
> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
> see, so we can learn from them). Of course there is a certain overhead in
> maintaining this all, and I don’t know if there are any roadblocks in the way
> BigCouch works today for implementing this.
> 
> Best
> Jan
> --
> 
> 
> 
> 
> 
> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
> 
>> I would also love to see _rev renamed, and I think it's a good
>> opportunity to flip around all the meta info as well. I'm still
>> partial to moving the relevant metadata into the headers, and no
>> longer including any _* fields in the doc, but I know there are
>> proponents on both sides of the coin there. The most recent proposal I
>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>> would like to see us move all metadata into headers or into the '_'
>> field, and rename 'rev'. There's a lot of code overlap for the two so
>> it seems like an opportune time to do it.
>> 
>> I wonder if it's reasonable to make the use of a '_' field or exposed
>> through headers configurable. I'm not sure it's a great idea to do so,
>> but it's at least worth thinking about.
>> 
>> Exposing conflicts by default is another thing I'm keen on. The
>> question is how to make it "fail" loudly so that client libraries
>> don't just think it's the document body. An aggressive approach send a
>> list of conflict revs rather than a doc object which will break all
>> existing parsers and require users to deal with. Then if you want a
>> particular rev, you'll need to specify it in the request.
>> 
>> We could also cleanup the API endpoints to make them more RESTful. IMO
>> things like _all_dbs and _all_docs should be the top level endpoints
>> and the current info endpoints moved to _info or some such.
>> 
>> Along the lines of API cleanup is the capabilities engine. I think
>> this would be a great thing to land, and if done properly could be a
>> self defining REST endpoint showing all the things the server is
>> capable of and how to reach them.
>> 
>> 
>> 
>> -Russell
>> 
>> 
>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>> 
>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>> <rn...@apache.org> wrote:
>>> Great point, +1 to just making that change on master right now.
>>> 
>>> B.
>>> 
>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>> 
>>>> I would like to see 'JSONP responses should be sent with a
>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>> change and the original issue is flying around in Jira for years.
>>>> 
>>>> Best,
>>>> Robert
>>>> 
>>>> 
>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>> 
>>>>> 
>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>> changes.
>>>>> 
>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>> version bump.
>>>>> 
>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>> change might be.
>>>>> 
>>>>> B.
>>>>> 
>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>> 
>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>> entirely from scratch.
>>>>>> 
>>>>>> -Joan
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>> To: dev@couchdb.apache.org
>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>> 
>>>>>> 
>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>> so very recently, so I will be pulling that work over soon).
>>>>>> 
>>>>>> A better view server protocol is also a great idea.
>>>>>> 
>>>>>> 
>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>> others design function are works differently from views. While it's
>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>> small steps in good direction.
>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>> 
>>>>>>> 
>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>> 
>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>> 
>>>>>>> Kind regards,
>>>>>>> Samuel
>>>>>> 
>>>>> 
>>>>> 
>>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Hans J Schroeder <hs...@cloudno.de>]

I am also -1. I would also be careful with API and JSON changes. At least there must be a compatibility switch.
Maybe we can consolidate all meta data in a "_" sub object, but rewrite the JSON docs on the fly in compatibility mode to the old structure.

When talking changes, I would recommend to make the "type" field a mandatory meta field and allow the indexer
to use it. This can dramatically reduce on-demand indexing time especially on big databases and 
databases with many new document writes.

Cheers

--
Hans    

On 18 Jul 2014, at 22:15 , Jan Lehnardt <ja...@apache.org> wrote:

> I’m major -1 on substantial API changes *just* because we are having some
> by necessity of getting BigCouch in.
> 
> The minor improvements mentioned previously in this thread sound reasonable,
> but changing the main JSON format seems like a rather bad idea as it will
> just break all clients. While the scenario is a little different, I’d like to
> avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
> 1.0 than Python 3 had over 2, but still, there is no need to make this harder
> for our users, if we don’t have to).
> 
> That said, we likely want to evolve the API at some point and I think we should
> nail down a strategy on *how* to do that, before getting into the details of
> what should change.
> 
> One option, and I haven’t thought this through, would be to use separate ports
> for a new API endpoint that we can evolve while keeping the current one. And
> we can do the deprecation and switch dance some time in the future. We could
> even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
> see, so we can learn from them). Of course there is a certain overhead in
> maintaining this all, and I don’t know if there are any roadblocks in the way
> BigCouch works today for implementing this.
> 
> Best
> Jan
> --
> 
> 
> 
> 
> 
> On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:
> 
>> I would also love to see _rev renamed, and I think it's a good
>> opportunity to flip around all the meta info as well. I'm still
>> partial to moving the relevant metadata into the headers, and no
>> longer including any _* fields in the doc, but I know there are
>> proponents on both sides of the coin there. The most recent proposal I
>> could find is to move all the metadata into a '_' field [1]. In 2.0 I
>> would like to see us move all metadata into headers or into the '_'
>> field, and rename 'rev'. There's a lot of code overlap for the two so
>> it seems like an opportune time to do it.
>> 
>> I wonder if it's reasonable to make the use of a '_' field or exposed
>> through headers configurable. I'm not sure it's a great idea to do so,
>> but it's at least worth thinking about.
>> 
>> Exposing conflicts by default is another thing I'm keen on. The
>> question is how to make it "fail" loudly so that client libraries
>> don't just think it's the document body. An aggressive approach send a
>> list of conflict revs rather than a doc object which will break all
>> existing parsers and require users to deal with. Then if you want a
>> particular rev, you'll need to specify it in the request.
>> 
>> We could also cleanup the API endpoints to make them more RESTful. IMO
>> things like _all_dbs and _all_docs should be the top level endpoints
>> and the current info endpoints moved to _info or some such.
>> 
>> Along the lines of API cleanup is the capabilities engine. I think
>> this would be a great thing to land, and if done properly could be a
>> self defining REST endpoint showing all the things the server is
>> capable of and how to reach them.
>> 
>> 
>> 
>> -Russell
>> 
>> 
>> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
>> 
>> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
>> <rn...@apache.org> wrote:
>>> Great point, +1 to just making that change on master right now.
>>> 
>>> B.
>>> 
>>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>>> 
>>>> I would like to see 'JSONP responses should be sent with a
>>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>>> change and the original issue is flying around in Jira for years.
>>>> 
>>>> Best,
>>>> Robert
>>>> 
>>>> 
>>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>>> 
>>>>> 
>>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>>> changes.
>>>>> 
>>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>>> version bump.
>>>>> 
>>>>> With that said then, I’m interested in hearing what else, besides the two
>>>>> merges, we feel we want to take on in our first major revision bump in
>>>>> approximately forever? At minimum, I would like to see a change that allows
>>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>>> change might be.
>>>>> 
>>>>> B.
>>>>> 
>>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>>> 
>>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>>> entirely from scratch.
>>>>>> 
>>>>>> -Joan
>>>>>> 
>>>>>> ----- Original Message -----
>>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>>> To: dev@couchdb.apache.org
>>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>>> 
>>>>>> 
>>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>>> so very recently, so I will be pulling that work over soon).
>>>>>> 
>>>>>> A better view server protocol is also a great idea.
>>>>>> 
>>>>>> 
>>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>>> space.ship.traveller@gmail.com> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>>> Our view server is compiles functions on each view index update
>>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>>> others design function are works differently from views. While it's
>>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>>> small steps in good direction.
>>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>>> 
>>>>>>> 
>>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>>> 
>>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>>> 
>>>>>>> Kind regards,
>>>>>>> Samuel
>>>>>> 
>>>>> 
>>>>> 
>>> 
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Jan Lehnardt <ja...@apache.org>]

I’m major -1 on substantial API changes *just* because we are having some
by necessity of getting BigCouch in.

The minor improvements mentioned previously in this thread sound reasonable,
but changing the main JSON format seems like a rather bad idea as it will
just break all clients. While the scenario is a little different, I’d like to
avoid a Python 3 kind of situation (I think CouchDB 2.0 has more to offer over
1.0 than Python 3 had over 2, but still, there is no need to make this harder
for our users, if we don’t have to).

That said, we likely want to evolve the API at some point and I think we should
nail down a strategy on *how* to do that, before getting into the details of
what should change.

One option, and I haven’t thought this through, would be to use separate ports
for a new API endpoint that we can evolve while keeping the current one. And
we can do the deprecation and switch dance some time in the future. We could
even try multiple competing APIs, even non HTTP APIs (all things, I’d love to
see, so we can learn from them). Of course there is a certain overhead in
maintaining this all, and I don’t know if there are any roadblocks in the way
BigCouch works today for implementing this.

Best
Jan
--





On 17 Jul 2014, at 21:03 , Russell Branca <ch...@apache.org> wrote:

> I would also love to see _rev renamed, and I think it's a good
> opportunity to flip around all the meta info as well. I'm still
> partial to moving the relevant metadata into the headers, and no
> longer including any _* fields in the doc, but I know there are
> proponents on both sides of the coin there. The most recent proposal I
> could find is to move all the metadata into a '_' field [1]. In 2.0 I
> would like to see us move all metadata into headers or into the '_'
> field, and rename 'rev'. There's a lot of code overlap for the two so
> it seems like an opportune time to do it.
> 
> I wonder if it's reasonable to make the use of a '_' field or exposed
> through headers configurable. I'm not sure it's a great idea to do so,
> but it's at least worth thinking about.
> 
> Exposing conflicts by default is another thing I'm keen on. The
> question is how to make it "fail" loudly so that client libraries
> don't just think it's the document body. An aggressive approach send a
> list of conflict revs rather than a doc object which will break all
> existing parsers and require users to deal with. Then if you want a
> particular rev, you'll need to specify it in the request.
> 
> We could also cleanup the API endpoints to make them more RESTful. IMO
> things like _all_dbs and _all_docs should be the top level endpoints
> and the current info endpoints moved to _info or some such.
> 
> Along the lines of API cleanup is the capabilities engine. I think
> this would be a great thing to land, and if done properly could be a
> self defining REST endpoint showing all the things the server is
> capable of and how to reach them.
> 
> 
> 
> -Russell
> 
> 
> [1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E
> 
> On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
> <rn...@apache.org> wrote:
>> Great point, +1 to just making that change on master right now.
>> 
>> B.
>> 
>> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>> 
>>> I would like to see 'JSONP responses should be sent with a
>>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>>> beside the two merges in the 2.0 release - it is a small, but breaking
>>> change and the original issue is flying around in Jira for years.
>>> 
>>> Best,
>>> Robert
>>> 
>>> 
>>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>> 
>>>> 
>>>> Since we follow semantic versioning, the only meaning behind naming our
>>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>>> changes.
>>>> 
>>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>>> version bump.
>>>> 
>>>> With that said then, I’m interested in hearing what else, besides the two
>>>> merges, we feel we want to take on in our first major revision bump in
>>>> approximately forever? At minimum, I would like to see a change that allows
>>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>>> change might be.
>>>> 
>>>> B.
>>>> 
>>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>> 
>>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>>> entirely from scratch.
>>>>> 
>>>>> -Joan
>>>>> 
>>>>> ----- Original Message -----
>>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>>> To: dev@couchdb.apache.org
>>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>> 
>>>>> 
>>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>>> so very recently, so I will be pulling that work over soon).
>>>>> 
>>>>> A better view server protocol is also a great idea.
>>>>> 
>>>>> 
>>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>>> space.ship.traveller@gmail.com> wrote:
>>>>> 
>>>>>> 
>>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>>> Our view server is compiles functions on each view index update
>>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>>> others design function are works differently from views. While it's
>>>>>>> good to change and improve query server protocol completely, this task
>>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>>> small steps in good direction.
>>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>> 
>>>>>> 
>>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>> 
>>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>> 
>>>>>> Kind regards,
>>>>>> Samuel
>>>>> 
>>>> 
>>>> 
>> 

Re: CouchDB 2.0: breaking the backward compatibility

[Joan Touzet <wo...@apache.org>]

Another wishlist item that I've wanted for ages is to fix the URL
namespace issue, to get us closer to other systems' REST-like
semantics. For instance:

  http://couchdb.atypical.net/joanpix

becomes

  http://couchdb.atypical.net/db/joanpix

In other words, we don't pollute the top URL scope level with
database names, it always is reserved for module/subsystem
references. This could be friendlier for plugins as well.
It might also help straighten out the _config namespace a bit.

----- Original Message -----
From: "Russell Branca" <ch...@apache.org>
To: "CouchDB Developers" <de...@couchdb.apache.org>
Sent: Thursday, July 17, 2014 3:03:01 PM
Subject: Re: CouchDB 2.0: breaking the backward compatibility

I would also love to see _rev renamed, and I think it's a good
opportunity to flip around all the meta info as well. I'm still
partial to moving the relevant metadata into the headers, and no
longer including any _* fields in the doc, but I know there are
proponents on both sides of the coin there. The most recent proposal I
could find is to move all the metadata into a '_' field [1]. In 2.0 I
would like to see us move all metadata into headers or into the '_'
field, and rename 'rev'. There's a lot of code overlap for the two so
it seems like an opportune time to do it.

I wonder if it's reasonable to make the use of a '_' field or exposed
through headers configurable. I'm not sure it's a great idea to do so,
but it's at least worth thinking about.

Exposing conflicts by default is another thing I'm keen on. The
question is how to make it "fail" loudly so that client libraries
don't just think it's the document body. An aggressive approach send a
list of conflict revs rather than a doc object which will break all
existing parsers and require users to deal with. Then if you want a
particular rev, you'll need to specify it in the request.

We could also cleanup the API endpoints to make them more RESTful. IMO
things like _all_dbs and _all_docs should be the top level endpoints
and the current info endpoints moved to _info or some such.

Along the lines of API cleanup is the capabilities engine. I think
this would be a great thing to land, and if done properly could be a
self defining REST endpoint showing all the things the server is
capable of and how to reach them.



-Russell


[1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E

On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
> Great point, +1 to just making that change on master right now.
>
> B.
>
> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>
>> I would like to see 'JSONP responses should be sent with a
>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>> beside the two merges in the 2.0 release - it is a small, but breaking
>> change and the original issue is flying around in Jira for years.
>>
>> Best,
>> Robert
>>
>>
>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>
>>>
>>> Since we follow semantic versioning, the only meaning behind naming our
>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>> changes.
>>>
>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>> version bump.
>>>
>>> With that said then, I’m interested in hearing what else, besides the two
>>> merges, we feel we want to take on in our first major revision bump in
>>> approximately forever? At minimum, I would like to see a change that allows
>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>> change might be.
>>>
>>> B.
>>>
>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>
>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>> entirely from scratch.
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>> To: dev@couchdb.apache.org
>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>
>>>>
>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>> so very recently, so I will be pulling that work over soon).
>>>>
>>>> A better view server protocol is also a great idea.
>>>>
>>>>
>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>> space.ship.traveller@gmail.com> wrote:
>>>>
>>>>>
>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>> Our view server is compiles functions on each view index update
>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>> others design function are works differently from views. While it's
>>>>>> good to change and improve query server protocol completely, this task
>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>> small steps in good direction.
>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>
>>>>>
>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>
>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>
>>>>> Kind regards,
>>>>> Samuel
>>>>
>>>
>>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Russell Branca <ch...@apache.org>]

I would also love to see _rev renamed, and I think it's a good
opportunity to flip around all the meta info as well. I'm still
partial to moving the relevant metadata into the headers, and no
longer including any _* fields in the doc, but I know there are
proponents on both sides of the coin there. The most recent proposal I
could find is to move all the metadata into a '_' field [1]. In 2.0 I
would like to see us move all metadata into headers or into the '_'
field, and rename 'rev'. There's a lot of code overlap for the two so
it seems like an opportune time to do it.

I wonder if it's reasonable to make the use of a '_' field or exposed
through headers configurable. I'm not sure it's a great idea to do so,
but it's at least worth thinking about.

Exposing conflicts by default is another thing I'm keen on. The
question is how to make it "fail" loudly so that client libraries
don't just think it's the document body. An aggressive approach send a
list of conflict revs rather than a doc object which will break all
existing parsers and require users to deal with. Then if you want a
particular rev, you'll need to specify it in the request.

We could also cleanup the API endpoints to make them more RESTful. IMO
things like _all_dbs and _all_docs should be the top level endpoints
and the current info endpoints moved to _info or some such.

Along the lines of API cleanup is the capabilities engine. I think
this would be a great thing to land, and if done properly could be a
self defining REST endpoint showing all the things the server is
capable of and how to reach them.



-Russell


[1] http://mail-archives.apache.org/mod_mbox/couchdb-dev/201312.mbox/%3C529DE44C.4090400@bigbluehat.com%3E

On Thu, Jul 17, 2014 at 2:14 AM, Robert Samuel Newson
<rn...@apache.org> wrote:
> Great point, +1 to just making that change on master right now.
>
> B.
>
> On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:
>
>> I would like to see 'JSONP responses should be sent with a
>> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
>> beside the two merges in the 2.0 release - it is a small, but breaking
>> change and the original issue is flying around in Jira for years.
>>
>> Best,
>> Robert
>>
>>
>> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
>>
>>>
>>> Since we follow semantic versioning, the only meaning behind naming our
>>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>>> changes.
>>>
>>> It’s for the CouchDB community as a whole to determine what is and isn’t
>>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>>> the 2.0 release, but they aren’t necessarily the only things. If they
>>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>>> version bump.
>>>
>>> With that said then, I’m interested in hearing what else, besides the two
>>> merges, we feel we want to take on in our first major revision bump in
>>> approximately forever? At minimum, I would like to see a change that allows
>>> us to use versions of spidermonkey released after 1.8.5, whatever that
>>> change might be.
>>>
>>> B.
>>>
>>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>>>
>>>> Improving the view server protocol is a great idea, but it is appropriate
>>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>>> entirely from scratch.
>>>>
>>>> -Joan
>>>>
>>>> ----- Original Message -----
>>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>>> To: dev@couchdb.apache.org
>>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>>>
>>>>
>>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>>> so very recently, so I will be pulling that work over soon).
>>>>
>>>> A better view server protocol is also a great idea.
>>>>
>>>>
>>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>>> space.ship.traveller@gmail.com> wrote:
>>>>
>>>>>
>>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>>> Our view server is compiles functions on each view index update
>>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>>> others design function are works differently from views. While it's
>>>>>> good to change and improve query server protocol completely, this task
>>>>>> requires more time to be done. We should have a least plan B to do
>>>>>> small steps in good direction.
>>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>>>
>>>>>
>>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>>>
>>>>> I welcome people to suggest improvements/changes/ideas.
>>>>>
>>>>> Kind regards,
>>>>> Samuel
>>>>
>>>
>>>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

Great point, +1 to just making that change on master right now.

B.

On 16 Jul 2014, at 22:35, Robert Kowalski <ro...@kowalski.gd> wrote:

> I would like to see 'JSONP responses should be sent with a
> "application/javascript"' (https://github.com/apache/couchdb/pull/236)
> beside the two merges in the 2.0 release - it is a small, but breaking
> change and the original issue is flying around in Jira for years.
> 
> Best,
> Robert
> 
> 
> 2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:
> 
>> 
>> Since we follow semantic versioning, the only meaning behind naming our
>> next release 2.0 and not 1.7 is that it contains backwards incompatible
>> changes.
>> 
>> It’s for the CouchDB community as a whole to determine what is and isn’t
>> in a release. Certainly merging in bigcouch and rcouch are a huge part of
>> the 2.0 release, but they aren’t necessarily the only things. If they
>> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
>> version bump.
>> 
>> With that said then, I’m interested in hearing what else, besides the two
>> merges, we feel we want to take on in our first major revision bump in
>> approximately forever? At minimum, I would like to see a change that allows
>> us to use versions of spidermonkey released after 1.8.5, whatever that
>> change might be.
>> 
>> B.
>> 
>> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>> 
>>> Improving the view server protocol is a great idea, but it is appropriate
>>> for a 2.0 timeframe? I would think it would make more sense in a 3.0
>>> timeframe, given 2.0 is all about merging forks, not writing new features
>>> entirely from scratch.
>>> 
>>> -Joan
>>> 
>>> ----- Original Message -----
>>> From: "Robert Samuel Newson" <rn...@apache.org>
>>> To: dev@couchdb.apache.org
>>> Sent: Sunday, July 13, 2014 8:52:40 AM
>>> Subject: Re: CouchDB 2.0: breaking the backward compatibility
>>> 
>>> 
>>> Adding mvcc for _security is a great idea (happily, Cloudant have done
>> so very recently, so I will be pulling that work over soon).
>>> 
>>> A better view server protocol is also a great idea.
>>> 
>>> 
>>> On 13 Jul 2014, at 13:13, Samuel Williams <
>> space.ship.traveller@gmail.com> wrote:
>>> 
>>>> 
>>>> On 13/07/14 23:47, Alexander Shorin wrote:
>>>>> Our view server is compiles functions on each view index update
>>>>> instead of reusing inner cache. This is because of out-dated protocol:
>>>>> others design function are works differently from views. While it's
>>>>> good to change and improve query server protocol completely, this task
>>>>> requires more time to be done. We should have a least plan B to do
>>>>> small steps in good direction.
>>>> As already suggested, here is my proposal for 2.0 view/query server:
>>>> 
>>>> 
>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>>>> 
>>>> I welcome people to suggest improvements/changes/ideas.
>>>> 
>>>> Kind regards,
>>>> Samuel
>>> 
>> 
>> 

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Kowalski <ro...@kowalski.gd>]

I would like to see 'JSONP responses should be sent with a
"application/javascript"' (https://github.com/apache/couchdb/pull/236)
beside the two merges in the 2.0 release - it is a small, but breaking
change and the original issue is flying around in Jira for years.

Best,
Robert


2014-07-13 22:17 GMT+02:00 Robert Samuel Newson <rn...@apache.org>:

>
> Since we follow semantic versioning, the only meaning behind naming our
> next release 2.0 and not 1.7 is that it contains backwards incompatible
> changes.
>
> It’s for the CouchDB community as a whole to determine what is and isn’t
> in a release. Certainly merging in bigcouch and rcouch are a huge part of
> the 2.0 release, but they aren’t necessarily the only things. If they
> hadn’t changed the API in incompatible ways, they wouldn’t cause a major
> version bump.
>
> With that said then, I’m interested in hearing what else, besides the two
> merges, we feel we want to take on in our first major revision bump in
> approximately forever? At minimum, I would like to see a change that allows
> us to use versions of spidermonkey released after 1.8.5, whatever that
> change might be.
>
> B.
>
> On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:
>
> > Improving the view server protocol is a great idea, but it is appropriate
> > for a 2.0 timeframe? I would think it would make more sense in a 3.0
> > timeframe, given 2.0 is all about merging forks, not writing new features
> > entirely from scratch.
> >
> > -Joan
> >
> > ----- Original Message -----
> > From: "Robert Samuel Newson" <rn...@apache.org>
> > To: dev@couchdb.apache.org
> > Sent: Sunday, July 13, 2014 8:52:40 AM
> > Subject: Re: CouchDB 2.0: breaking the backward compatibility
> >
> >
> > Adding mvcc for _security is a great idea (happily, Cloudant have done
> so very recently, so I will be pulling that work over soon).
> >
> > A better view server protocol is also a great idea.
> >
> >
> > On 13 Jul 2014, at 13:13, Samuel Williams <
> space.ship.traveller@gmail.com> wrote:
> >
> >>
> >> On 13/07/14 23:47, Alexander Shorin wrote:
> >>> Our view server is compiles functions on each view index update
> >>> instead of reusing inner cache. This is because of out-dated protocol:
> >>> others design function are works differently from views. While it's
> >>> good to change and improve query server protocol completely, this task
> >>> requires more time to be done. We should have a least plan B to do
> >>> small steps in good direction.
> >> As already suggested, here is my proposal for 2.0 view/query server:
> >>
> >>
> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
> >>
> >> I welcome people to suggest improvements/changes/ideas.
> >>
> >> Kind regards,
> >> Samuel
> >
>
>

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

Since we follow semantic versioning, the only meaning behind naming our next release 2.0 and not 1.7 is that it contains backwards incompatible changes.

It’s for the CouchDB community as a whole to determine what is and isn’t in a release. Certainly merging in bigcouch and rcouch are a huge part of the 2.0 release, but they aren’t necessarily the only things. If they hadn’t changed the API in incompatible ways, they wouldn’t cause a major version bump.

With that said then, I’m interested in hearing what else, besides the two merges, we feel we want to take on in our first major revision bump in approximately forever? At minimum, I would like to see a change that allows us to use versions of spidermonkey released after 1.8.5, whatever that change might be.

B.

On 13 Jul 2014, at 20:31, Joan Touzet <wo...@apache.org> wrote:

> Improving the view server protocol is a great idea, but it is appropriate
> for a 2.0 timeframe? I would think it would make more sense in a 3.0
> timeframe, given 2.0 is all about merging forks, not writing new features
> entirely from scratch.
> 
> -Joan
> 
> ----- Original Message -----
> From: "Robert Samuel Newson" <rn...@apache.org>
> To: dev@couchdb.apache.org
> Sent: Sunday, July 13, 2014 8:52:40 AM
> Subject: Re: CouchDB 2.0: breaking the backward compatibility
> 
> 
> Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).
> 
> A better view server protocol is also a great idea.
> 
> 
> On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:
> 
>> 
>> On 13/07/14 23:47, Alexander Shorin wrote:
>>> Our view server is compiles functions on each view index update
>>> instead of reusing inner cache. This is because of out-dated protocol:
>>> others design function are works differently from views. While it's
>>> good to change and improve query server protocol completely, this task
>>> requires more time to be done. We should have a least plan B to do
>>> small steps in good direction.
>> As already suggested, here is my proposal for 2.0 view/query server:
>> 
>> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
>> 
>> I welcome people to suggest improvements/changes/ideas.
>> 
>> Kind regards,
>> Samuel
> 

Re: CouchDB 2.0: breaking the backward compatibility

[Joan Touzet <wo...@apache.org>]

Improving the view server protocol is a great idea, but it is appropriate
for a 2.0 timeframe? I would think it would make more sense in a 3.0
timeframe, given 2.0 is all about merging forks, not writing new features
entirely from scratch.

-Joan

----- Original Message -----
From: "Robert Samuel Newson" <rn...@apache.org>
To: dev@couchdb.apache.org
Sent: Sunday, July 13, 2014 8:52:40 AM
Subject: Re: CouchDB 2.0: breaking the backward compatibility


Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).

A better view server protocol is also a great idea.


On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:

> 
> On 13/07/14 23:47, Alexander Shorin wrote:
>> Our view server is compiles functions on each view index update
>> instead of reusing inner cache. This is because of out-dated protocol:
>> others design function are works differently from views. While it's
>> good to change and improve query server protocol completely, this task
>> requires more time to be done. We should have a least plan B to do
>> small steps in good direction.
> As already suggested, here is my proposal for 2.0 view/query server:
> 
> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
> 
> I welcome people to suggest improvements/changes/ideas.
> 
> Kind regards,
> Samuel

Re: CouchDB 2.0: breaking the backward compatibility

[Robert Samuel Newson <rn...@apache.org>]

Adding mvcc for _security is a great idea (happily, Cloudant have done so very recently, so I will be pulling that work over soon).

A better view server protocol is also a great idea.


On 13 Jul 2014, at 13:13, Samuel Williams <sp...@gmail.com> wrote:

> 
> On 13/07/14 23:47, Alexander Shorin wrote:
>> Our view server is compiles functions on each view index update
>> instead of reusing inner cache. This is because of out-dated protocol:
>> others design function are works differently from views. While it's
>> good to change and improve query server protocol completely, this task
>> requires more time to be done. We should have a least plan B to do
>> small steps in good direction.
> As already suggested, here is my proposal for 2.0 view/query server:
> 
> https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit
> 
> I welcome people to suggest improvements/changes/ideas.
> 
> Kind regards,
> Samuel

Re: CouchDB 2.0: breaking the backward compatibility

[Samuel Williams <sp...@gmail.com>]

On 13/07/14 23:47, Alexander Shorin wrote:
> Our view server is compiles functions on each view index update
> instead of reusing inner cache. This is because of out-dated protocol:
> others design function are works differently from views. While it's
> good to change and improve query server protocol completely, this task
> requires more time to be done. We should have a least plan B to do
> small steps in good direction.
As already suggested, here is my proposal for 2.0 view/query server:

https://docs.google.com/document/d/1JtfvCpNB9pRQyLhS5KkkEdJ-ghSCv89xnw5HDMTCsp8/edit

I welcome people to suggest improvements/changes/ideas.

Kind regards,
Samuel