App layer on top of replication

[Eduardo Scoz <ed...@gmail.com>]

Hi guys,

Is there any way to put an app server between two CouchDB instances to
manage security during replication?

I'm working on a mobile app using TouchDB (iOS and Android), and I'm
replicating the data back to the server using the normal CouchDB
replication. In my use case, user creates new documents locally on the
device, and syncs the DB with the host to push changes and get new
documents. Filtered replication works well for getting only important docs
and validate_doc_update is enough to prevent unauthorized writes, but that
still leaves the database open for anybody with an ID to read other user's
docs.

My idea is to put CouchDB behind something like Nginx or NodeJS that would
intercept direct calls to the database, but still let replication work (as
described above). all_docs replication would also have to be disabled, I
guess.

has anyone done anything like this? Any pointers on how one would make it?

Thanks!

Eduardo

Re: App layer on top of replication

[Eduardo Scoz <ed...@gmail.com>]

Thanks for the reply Jens. I'll start working on it probably this weekend,
and lets see what comes out of it.

Hopefully it won't be hard to modify a simple nodejs proxy to do the basics
of what I want, but you never know (and I'm not that great with js)..



On Fri, Sep 7, 2012 at 5:05 PM, Jens Alfke <je...@couchbase.com> wrote:

>
> On Sep 6, 2012, at 4:54 PM, Eduardo Scoz <ed...@gmail.com> wrote:
>
> > Is there a list somewhere of all the possible requests that couchdb
> accepts
> > (like _changes, _all_docs, etc)? Blocking all requests by default and
> > making only the few things I would like to have available could be a
> > possibility.
>
> Not really. The reference I usually use is this one in the wiki[1], but
> it’s incomplete; I’ve added details as I discover them, but I’m sure there
> are other things missing.
>
> Your approach of whitelisting might be the best way to do it, especially
> since the db on the server will only be accessed by the TouchDB replicator,
> which is fairly predictable in the subset of API calls that it makes.
>
> I’m interested to see whether this works for you, because this would be
> useful to other people too. The Syncpoint database-per-user approach is
> IMHO cleaner and more flexible, but will take a fair bit of doing to
> implement properly, and JChris hasn’t had time to devote to it lately.
>
> —Jens
>
> [1]: http://wiki.apache.org/couchdb/Complete_HTTP_API_Reference

Re: App layer on top of replication

[Jens Alfke <je...@couchbase.com>]

On Sep 6, 2012, at 4:54 PM, Eduardo Scoz <ed...@gmail.com> wrote:

> Is there a list somewhere of all the possible requests that couchdb accepts
> (like _changes, _all_docs, etc)? Blocking all requests by default and
> making only the few things I would like to have available could be a
> possibility.

Not really. The reference I usually use is this one in the wiki[1], but it’s incomplete; I’ve added details as I discover them, but I’m sure there are other things missing.

Your approach of whitelisting might be the best way to do it, especially since the db on the server will only be accessed by the TouchDB replicator, which is fairly predictable in the subset of API calls that it makes.

I’m interested to see whether this works for you, because this would be useful to other people too. The Syncpoint database-per-user approach is IMHO cleaner and more flexible, but will take a fair bit of doing to implement properly, and JChris hasn’t had time to devote to it lately.

—Jens

[1]: http://wiki.apache.org/couchdb/Complete_HTTP_API_Reference

Re: App layer on top of replication

[Eduardo Scoz <ed...@gmail.com>]

Hi Jens,

thanks for the reply.

Is there a list somewhere of all the possible requests that couchdb accepts
(like _changes, _all_docs, etc)? Blocking all requests by default and
making only the few things I would like to have available could be a
possibility.

A lot of data gets shared between the users in my case. A good analogy
would be an outlook calendar server, where users can have their private
events, but they can also share events with an limited number of people.
Couch (with filtered replication and validate_doc_update) gets me 99%
there, but it fails on the security aspect, because all users would have
access to all private events as well.





On Thu, Sep 6, 2012 at 6:05 PM, Jens Alfke <je...@couchbase.com> wrote:

>
> On Sep 6, 2012, at 1:41 PM, Eduardo Scoz <eduardoscoz@gmail.com<mailto:
> eduardoscoz@gmail.com>> wrote:
>
> My idea is to put CouchDB behind something like Nginx or NodeJS that would
> intercept direct calls to the database, but still let replication work (as
> described above). all_docs replication would also have to be disabled, I
> guess.
>
> What makes this tricky is that document contents leak out through other
> API calls besides direct GETs of a doc URL. For instance, you can look at
> all documents by getting the _changes feed or _all_docs, and of course
> querying views. Your proxy layer would have to be aware of these — for
> example it would have to block any requests for _changes unless they
> specified a filter that restricted the results to docs that user can see
> (or maybe it could rewrite the URL to add such a filter?)
>
> I think this is do-able but it’s the sort of thing where I’d worry that
> someone will come up with some creative use of the API to get around it.
>
> A proposed solution by the couchbase guys is to have multiple databases,
> one for each user, and then use replication to share the data, but this
> seems unmanageable to me once you get thousands of users syncing docs
> around.
>
> It depends on your data model. At one extreme, if users’ data is entirely
> private, the user databases will all be independent and there’s no
> overhead. How much data is shared between your users?
>
> —Jens
>

Re: App layer on top of replication

[Jens Alfke <je...@couchbase.com>]

On Sep 6, 2012, at 1:41 PM, Eduardo Scoz <ed...@gmail.com>> wrote:

My idea is to put CouchDB behind something like Nginx or NodeJS that would
intercept direct calls to the database, but still let replication work (as
described above). all_docs replication would also have to be disabled, I
guess.

What makes this tricky is that document contents leak out through other API calls besides direct GETs of a doc URL. For instance, you can look at all documents by getting the _changes feed or _all_docs, and of course querying views. Your proxy layer would have to be aware of these — for example it would have to block any requests for _changes unless they specified a filter that restricted the results to docs that user can see (or maybe it could rewrite the URL to add such a filter?)

I think this is do-able but it’s the sort of thing where I’d worry that someone will come up with some creative use of the API to get around it.

A proposed solution by the couchbase guys is to have multiple databases,
one for each user, and then use replication to share the data, but this
seems unmanageable to me once you get thousands of users syncing docs
around.

It depends on your data model. At one extreme, if users’ data is entirely private, the user databases will all be independent and there’s no overhead. How much data is shared between your users?

—Jens

Re: App layer on top of replication

[Eduardo Scoz <ed...@gmail.com>]

On Thu, Sep 6, 2012 at 4:04 PM, Dave Cottlehuber <dc...@jsonified.com> wrote:

> On 6 September 2012 22:41, Eduardo Scoz <ed...@gmail.com> wrote:
> > Hi guys,
> >
> > Is there any way to put an app server between two CouchDB instances to
> > manage security during replication?
> >
> > I'm working on a mobile app using TouchDB (iOS and Android), and I'm
> > replicating the data back to the server using the normal CouchDB
> > replication. In my use case, user creates new documents locally on the
> > device, and syncs the DB with the host to push changes and get new
> > documents. Filtered replication works well for getting only important
> docs
> > and validate_doc_update is enough to prevent unauthorized writes, but
> that
> > still leaves the database open for anybody with an ID to read other
> user's
> > docs.
> >
> > My idea is to put CouchDB behind something like Nginx or NodeJS that
> would
> > intercept direct calls to the database, but still let replication work
> (as
> > described above). all_docs replication would also have to be disabled, I
> > guess.
> >
> > has anyone done anything like this? Any pointers on how one would make
> it?
> >
> > Thanks!
> >
> > Eduardo
>
> Hi Eduardo,
>
> Security is enforced at the DB level in CouchDB. Does your app architecture
> allow this?
>
> Fundamentally, the replicator is a clever, optimised HTTP client, so
> blocking all except
> the replicator will be very difficult, if not impossible.
>
> Any other ideas out there?
>
> A+
> Dave
>

My problem with the way couch implements security is that there's no
per-doc read validation, only on writes. A user either have full read
access, or no access at all (please correct me if I'm wrong).

A proposed solution by the couchbase guys is to have multiple databases,
one for each user, and then use replication to share the data, but this
seems unmanageable to me once you get thousands of users syncing docs
around.

The replicator also uses GETs on the documents to get their revisions,
though, no? I know it doesn't do it for new documents, but the filtered
replication would not cause that situation, I guess..

Thanks Dave!

Re: App layer on top of replication

[Dave Cottlehuber <dc...@jsonified.com>]

On 6 September 2012 22:41, Eduardo Scoz <ed...@gmail.com> wrote:
> Hi guys,
>
> Is there any way to put an app server between two CouchDB instances to
> manage security during replication?
>
> I'm working on a mobile app using TouchDB (iOS and Android), and I'm
> replicating the data back to the server using the normal CouchDB
> replication. In my use case, user creates new documents locally on the
> device, and syncs the DB with the host to push changes and get new
> documents. Filtered replication works well for getting only important docs
> and validate_doc_update is enough to prevent unauthorized writes, but that
> still leaves the database open for anybody with an ID to read other user's
> docs.
>
> My idea is to put CouchDB behind something like Nginx or NodeJS that would
> intercept direct calls to the database, but still let replication work (as
> described above). all_docs replication would also have to be disabled, I
> guess.
>
> has anyone done anything like this? Any pointers on how one would make it?
>
> Thanks!
>
> Eduardo

Hi Eduardo,

Security is enforced at the DB level in CouchDB. Does your app architecture
allow this?

Fundamentally, the replicator is a clever, optimised HTTP client, so
blocking all except
the replicator will be very difficult, if not impossible.

Any other ideas out there?

A+
Dave