Any way to explicitly skip design docs in replication?

[Jens Alfke <je...@couchbase.com>]

Replicating design docs seems to be problematic for native apps (or just non-CouchApps). We’ve run into several issues caused by it, in Couchbase Mobile:

PROBLEMS:
* If you don’t have admin privileges to a remote DB you’re pushing to, you’ll get an error message spammed to the log for every design doc in the local database, which makes it look like the replication is failing. There have already been several posts to the mobile-couchbase mailing list by worried developers noticing this message in their console output and wondering what’s wrong.

* If you do have admin privileges, a design doc pushed to a different server may not work there due to varying language support. For example, I can write functions in Erlang in my local server, but they won’t be allowed to run on IrisCouch [I think]. Similarly, Couchbase Mobile for iOS now supports functions that call into native code, but the “objc” language ID isn’t recognized by any other server. This makes the resulting functions not work.

* An especially bad corollary of the above is that if you replicate a design document containing a _validation_ function in a language that’s unsupported at the destination, the destination database will now *refuse to update any documents*. This makes the rest of the replication fail, and generally breaks the database completely until someone uses Futon or curl to delete or edit the offending design doc.

* We’ve also run into issues where two independently developed native apps that operate on the same data set each create a design doc for their views … and unintentionally give those design docs the same name. This then results in a conflict whenever these apps both try to replicate with the same database. (This is happening with our iOS and Android ‘Grocery Sync’ apps.) In this case it’s obvious that the design docs are purely part of the app implementation, not part of the data set, and should never be replicated.

* A similar thing happens if two versions of a native app sync to the same database, and in version 2 the design document was changed in a way that’s incompatible with version 1. After bidirectional syncing, one of the apps is likely to be broken since the ‘winning’ version of the design doc will be the one that it’s not compatible with.

SUGGESTION:
Due to this, I would like replication of design docs to be optional. I suggest adding something like a “designdocs”:false property in the replication’s JSON configuration.

Yes, it is possible to write your own filter function that skips design docs. But this then has to be implemented in every app; I don’t think there’s a way to factor this out so it can be handled by a framework (like my CouchCocoa) without the app developer having to do the work.

Comments? Has this been brought up before? Is there a ticket for it or should I file one?

—Jens

Re: Any way to explicitly skip design docs in replication?

[Nathan Vander Wilt <na...@calftrail.com>]

On Nov 30, 2011, at 8:04 AM, Jens Alfke wrote:
> 
> On Nov 30, 2011, at 6:18 AM, Robert Newson wrote:
> 
>> We could add another built-in filter to complement _design (which only
>> includes design docs), _not_design (or something less silly than that
>> even). Would be trivial and would avoid this class of problems.
> 
> I think this would work well.

While I don't have any other particularly compelling use cases for it (presently), one of my first questions when looking at the filtered replication API was "is there a way to invert the filter function"?

So IMO something less silly than "_not_design" would be something like a generic "invert_filter" parameter that would flip the return value of any specified filter function.

thanks,
-natevw

Re: Any way to explicitly skip design docs in replication?

[Jens Alfke <je...@couchbase.com>]

On Nov 30, 2011, at 6:18 AM, Robert Newson wrote:

> We could add another built-in filter to complement _design (which only
> includes design docs), _not_design (or something less silly than that
> even). Would be trivial and would avoid this class of problems.

I think this would work well.

> As for treating a validate_doc_update in an unsupported language as if
> it's not there, I'm hugely -1 on that, what a giant hole in our
> security model.

Agreed.

> a 400 Bad Request response for invalid ddocs is a
> great idea. Adam independently suggested the same for map/reduce/etc
> functions that don't *compile*, and this adds even more value to that
> notion.

An error at PUT time for ddocs with syntax errors would be great — in my brief experience I’ve seen this come up several times as a source of mysterious errors for newbies.

—Jens

Re: Any way to explicitly skip design docs in replication?

[Robert Newson <rn...@apache.org>]

We could add another built-in filter to complement _design (which only
includes design docs), _not_design (or something less silly than that
even). Would be trivial and would avoid this class of problems.

As for treating a validate_doc_update in an unsupported language as if
it's not there, I'm hugely -1 on that, what a giant hole in our
security model. a 400 Bad Request response for invalid ddocs is a
great idea. Adam independently suggested the same for map/reduce/etc
functions that don't *compile*, and this adds even more value to that
notion.

B.

On 30 November 2011 00:08, Jason Smith <jh...@iriscouch.com> wrote:
> Hi, Jens. You raise thought-provoking questions, as usual! IMHO,
> extending the "standard library" of prefab filter functions sounds
> nice.
>
> Also, Adam proposed a similar feature to ignore validation functions
> on the target couch. You may want to reread it and get some ideas.
> Submitted without commentary: you can find the thread by adding
> "worldview" to your search. :)
>
> On Wed, Nov 30, 2011 at 3:49 AM, Jens Alfke <je...@couchbase.com> wrote:
>> Replicating design docs seems to be problematic for native apps (or just non-CouchApps). We’ve run into several issues caused by it, in Couchbase Mobile:
>>
>> PROBLEMS:
>> * If you don’t have admin privileges to a remote DB you’re pushing to, you’ll get an error message spammed to the log for every design doc in the local database, which makes it look like the replication is failing. There have already been several posts to the mobile-couchbase mailing list by worried developers noticing this message in their console output and wondering what’s wrong.
>>
>> * If you do have admin privileges, a design doc pushed to a different server may not work there due to varying language support. For example, I can write functions in Erlang in my local server, but they won’t be allowed to run on IrisCouch [I think]. Similarly, Couchbase Mobile for iOS now supports functions that call into native code, but the “objc” language ID isn’t recognized by any other server. This makes the resulting functions not work.
>>
>> * An especially bad corollary of the above is that if you replicate a design document containing a _validation_ function in a language that’s unsupported at the destination, the destination database will now *refuse to update any documents*. This makes the rest of the replication fail, and generally breaks the database completely until someone uses Futon or curl to delete or edit the offending design doc.
>
> This one strikes me as a CouchDB bug independent of the other
> problems. CouchDB knows the languages it supports. When Couch receives
> a design document with an unsupported language, it seems like it
> should do one of two things:
>
> a) Reject the document as invalid (similar to how it would reject
> invalid JSON syntax)
> b) Accept the document and no-op the validations. (Seems more sensible)
>
> --
> Iris Couch
>

Re: Any way to explicitly skip design docs in replication?

[Jason Smith <jh...@iriscouch.com>]

Hi, Jens. You raise thought-provoking questions, as usual! IMHO,
extending the "standard library" of prefab filter functions sounds
nice.

Also, Adam proposed a similar feature to ignore validation functions
on the target couch. You may want to reread it and get some ideas.
Submitted without commentary: you can find the thread by adding
"worldview" to your search. :)

On Wed, Nov 30, 2011 at 3:49 AM, Jens Alfke <je...@couchbase.com> wrote:
> Replicating design docs seems to be problematic for native apps (or just non-CouchApps). We’ve run into several issues caused by it, in Couchbase Mobile:
>
> PROBLEMS:
> * If you don’t have admin privileges to a remote DB you’re pushing to, you’ll get an error message spammed to the log for every design doc in the local database, which makes it look like the replication is failing. There have already been several posts to the mobile-couchbase mailing list by worried developers noticing this message in their console output and wondering what’s wrong.
>
> * If you do have admin privileges, a design doc pushed to a different server may not work there due to varying language support. For example, I can write functions in Erlang in my local server, but they won’t be allowed to run on IrisCouch [I think]. Similarly, Couchbase Mobile for iOS now supports functions that call into native code, but the “objc” language ID isn’t recognized by any other server. This makes the resulting functions not work.
>
> * An especially bad corollary of the above is that if you replicate a design document containing a _validation_ function in a language that’s unsupported at the destination, the destination database will now *refuse to update any documents*. This makes the rest of the replication fail, and generally breaks the database completely until someone uses Futon or curl to delete or edit the offending design doc.

This one strikes me as a CouchDB bug independent of the other
problems. CouchDB knows the languages it supports. When Couch receives
a design document with an unsupported language, it seems like it
should do one of two things:

a) Reject the document as invalid (similar to how it would reject
invalid JSON syntax)
b) Accept the document and no-op the validations. (Seems more sensible)

-- 
Iris Couch