You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Jan Lehnardt <ja...@apache.org> on 2019/02/17 14:25:50 UTC
[DISCUSS] Per-doc access control
Hi Everyone,
I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
You can check out my branch here:
https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
It is very much work in progress, but it is far enough along to warrant discussion.
The main point of this branch is to show all the places that we would need to change to support the proposal.
Things I’ve left for later:
- currently only the first element in the _access array is used. Our and/or syntax can be added later.
- building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
- pretty HTTP error handling
- tests except for a tiny shell script 😇
Implementation notes:
You create a database with the _access feature turned on like so: PUT /db?access=true
I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
* * *
The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
* * *
The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
* * *
Open questions:
- The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
- Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
- building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
* * *
I’ll leave this here for now, I’m sure there are a few more things to consider.
I’d love to hear any and all feedback you might have. Especially if anything is unclear.
Best
Jan
—
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ma...@jan.io>.
One addition, the slotting in of _access into existing security mechanisms is as follows:
1. Check if a user is in _security
2. If yes, check it user is in _access (modulo read/write)
3. If yes, does the doc update pass any globally defined VDUs
4. If yes, operation can proceed.
Cheers
Jan
—
> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hey all,
>
> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>
> My guiding principles so far are:
>
> 1. Make the API intuitive, things should work like they look like they should work like.
> 2. The default should never be that a resources is accidentally left accessible to the public.
> 3. This should work as a natural extension to the existing security features*.
>
> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>
>
> ## Database Creation and Default Behaviours
>
> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>
> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>
> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>
> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>
> * * *
>
>
> ## Document Creation and Access Control
>
> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>
> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>
> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
>
> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>
> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well. We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>
> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>
> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>
> * * *
>
>
> ## _revs_diff
>
> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>
> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>
> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>
> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>
> * * *
>
>
> ## Partitioned Databases
>
> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>
> * * *
>
>
> ## Scenarios that _access should help with.
>
> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>
>
> ### Scenario 1: db-per-user
>
> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>
> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>
> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>
> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>
> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>
>
> ### Scenario 2: db-per-user + Sharing
>
> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>
> #### 2.1: The Todo List
>
> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>
> #### 2.2: The Chat/Forum/Newsgroup
>
> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>
> #### 2.3: The Corporate Hierarchy
>
> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>
>
> ### 3: Multiple Apps
>
> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>
> I’ll be referring to these scenarios down the line.
>
> * * *
>
>
> ## Design Docs
>
> ### Admin
>
> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>
> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>
>
> ### User
>
> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>
> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>
> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>
> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>
> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs, how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>
> We could say:
>
> a) we don’t support scenario 3.
> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>
> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>
> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>
>
> ## Read and Write Access
>
> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>
> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>
> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
>
> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []} or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>
> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>
>
> * * *
>
>
> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>
> Thanks for reading this far and I’m looking forward to your feedback.
>
>
> Best,
> Jan “_access” Lehnardt
> —
>
>
>
>
>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Hi Everyone,
>>
>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>
>> You can check out my branch here:
>>
>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>
>> It is very much work in progress, but it is far enough along to warrant discussion.
>>
>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>
>> Things I’ve left for later:
>>
>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>> - pretty HTTP error handling
>> - tests except for a tiny shell script 😇
>>
>> Implementation notes:
>>
>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>
>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>
>> * * *
>>
>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>
>> * * *
>>
>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>
>> * * *
>>
>> Open questions:
>>
>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>
>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>
>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>
>> * * *
>>
>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>
>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>
>> Best
>> Jan
>> —
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
Hi all,
I have a first RFC up here: https://github.com/apache/couchdb-documentation/pull/424
The goal of this is to have a concrete scope of changes written
down and have it be a minimal set to implement while being useful.
Specifically, this proposal defers to later:
* per-access views
* differentiation between read and write access for documents
* sharing individual documents between multiple users or groups
The feature design so far aims to allow the addition of the above
at a later point. Specifically, per-access views and read/write
differentiation should not be too hard to add, but doc sharing
might better be left for a FoundationDB future.
I’d like a thorough review and comments that specifically address
any holes in the proposal. In particular, I’m interested to hear
if there are any showstoppers in there that a common db-per-user
setup could not migrate over to this model.
Best
Jan
—
> On 4. Apr 2019, at 10:45, Jan Lehnardt <ja...@apache.org> wrote:
>
> Thanks for your initial comments.
>
>> On 3. Apr 2019, at 23:07, Adam Kocoloski <ko...@apache.org> wrote:
>>
>> I’m also in favor of dropping Scenario 3.
>>
>> One topic we may have discussed in the past but I wanted to close out here: in the relational database world it’s not uncommon to use materialized views as an access control mechanism to selectively expose contents of a table to clients who cannot access the table directly. Does the current thinking on _access for views support that use case? Can we build a view using a set of roles inherited from the user who created the design doc, but then turn around and set the _access on the view itself to a less-restrictive set?
>
> 3 minutes thinking it over didn’t reveal any particular problems with this feature, aside from include_docs not working as expected, which might be an okay trade-off for now. But could be included later.
>
>
>> On the _revs_diff topic — I’m not all that concerned about users trying to guess revision IDs that exist on the server, and then reverse-engineer the contents of the existing revisions. Maybe I ought to be.
>
> I’m not particularly worried, but it is at least a theoretical situation where our user’s can be caught with their pants down when they didn’t expect it. All I want to make sure is to document this properly. C.f. git where if you get access to a repo, you get the whole history, not just the state from where you started having access.
>
> Best
> Jan
> —
>
>>
>> On a somewhat-related note, I have had conversations before with folks who are keen to adopt these sorts of fine-grained access control systems who said they actually prefer to have a 403 Forbidden response list the set of privileges that would be sufficient to access the resource. I found this surprising, but I guess it comes down to a user needing to figure out what kind of security exception to apply for in order to make progress with some data analysis. I think this is a topic on which we could make a fairly late-binding decision — or even have it as a configurable option.
>>
>> I could definitely see the base Scenario 1 (single _access labels) landing ahead of the more-complex sharing models.
>>
>> I haven’t had a chance to take a deep look at the code but the design seems good and thoughtful, and I definitely like the focus on the use cases.
>>
>> Adam
>>
>>> On Mar 14, 2019, at 11:21 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> My replies now inline.
>>>
>>>> On 14. Mar 2019, at 16:13, Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> I received some notes privately from Gregor Martynus, which I’m reproducing here in email thread form. This email is all Gregor’s notes, my next email is my replies to them.
>>>>
>>>>> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>
>>>>> Hey all,
>>>>>
>>>>> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>>>>>
>>>>> My guiding principles so far are:
>>>>>
>>>>> 1. Make the API intuitive, things should work like they look like they should work like.
>>>>> 2. The default should never be that a resources is accidentally left accessible to the public.
>>>>> 3. This should work as a natural extension to the existing security features*.
>>>>>
>>>>> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>>>>>
>>>>>
>>>>> ## Database Creation and Default Behaviours
>>>>>
>>>>> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>>>>>
>>>>> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>>>>>
>>>>> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>>>>>
>>>>> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> ## Document Creation and Access Control
>>>>>
>>>>> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>>>>>
>>>>> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>>>>>
>>>>> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
>>>>
>>>> I prefer being explicit.
>>>>
>>>>
>>>>>
>>>>> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>>>>>
>>>>> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well.
>>>>
>>>> We could also provide tooling for migrations?
>>>
>>> I’d love tooling, but we’d have to make sure we can do it correctly for a big number of use-cases. For the acceptance of this change, I’d make “documenting a migration path for db-per-user setups” a MUST have, and any code that helps with that a nice to have.
>>>
>>>>
>>>>
>>>>> We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>>>>>
>>>>> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>>>>>
>>>>> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> ## _revs_diff
>>>>>
>>>>> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>>>>>
>>>>> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>>>>>
>>>>> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>>>>>
>>>>> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> ## Partitioned Databases
>>>>>
>>>>> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> ## Scenarios that _access should help with.
>>>>>
>>>>> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>>>>>
>>>>>
>>>>> ### Scenario 1: db-per-user
>>>>>
>>>>> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>>>>>
>>>>> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>>>>>
>>>>> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>>>>>
>>>>> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>>>>>
>>>>> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>>>>>
>>>>>
>>>>> ### Scenario 2: db-per-user + Sharing
>>>>
>>>> One scenario we should address is how stopping to share would work when documents are continuously replicated, e.g. to a client for offline usage. My understanding is that for the person who’s access to documents got revoked does not get _changes update telling them that their access got removed, it would be up to the application developer to implement some kind of "notification" meta documents. Unless you have a better idea?
>>>
>>> Since we now have a purge API as well, we could treat an un-share as a purge for clients, and they can decide what to do with it.
>>>
>>> Alternatively, we need to make breaking changes to _changes feed, maybe we can hide that behind an opt-in flag, like “/db/_changes?access=true”, and then we can send new rows like:
>>>
>>> {seq: XYZ, id: abc, rev:4-YYY, _revoked: true} or somesuch.
>>>
>>>
>>>>
>>>>>
>>>>> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>>>>>
>>>>> #### 2.1: The Todo List
>>>>>
>>>>> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>>>>>
>>>>> #### 2.2: The Chat/Forum/Newsgroup
>>>>>
>>>>> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>>>>>
>>>>> #### 2.3: The Corporate Hierarchy
>>>>>
>>>>> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>>>>>
>>>>>
>>>>> ### 3: Multiple Apps
>>>>>
>>>>> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>>>>>
>>>>> I’ll be referring to these scenarios down the line.
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> ## Design Docs
>>>>>
>>>>> ### Admin
>>>>>
>>>>> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>>>>>
>>>>> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>>>>>
>>>>>
>>>>> ### User
>>>>>
>>>>> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>>>>>
>>>>> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>>>>>
>>>>> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>>>>>
>>>>> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>>>>>
>>>>> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs,
>>>>
>>>> I can’t think of a db-per-user scenario where each user DB would have a different validate_doc_update method? It would be the same method with access to the user context, the DBs security setting and the document, so it would act differently for different users, but using the same code.
>>>
>>> They wouldn’t be different, but if we were do replicate 1000 db-per-user design docs into a single database, as per today’s semantics, we’d have to run 1000 VDUs on each doc update.
>>>
>>>>
>>>>> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>>>>>
>>>>> We could say:
>>>>>
>>>>> a) we don’t support scenario 3.
>>>>
>>>> +1, I think it would make our lives easier in general if we don’t recommend to share the same CouchDB for multiple apps. At least I don’t see a reason to do that at this point.
>>>
>>> I think I like this best, too, but I’d like to hear from others as well.
>>>
>>>
>>> Best
>>> Jan
>>> —
>>>>
>>>>> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
>>>>> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>>>>>
>>>>> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>>>>>
>>>>> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>>>>>
>>>>>
>>>>> ## Read and Write Access
>>>>>
>>>>> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>>>>>
>>>>> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>>>>>
>>>>> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
>>>>
>>>> Yes!
>>>>
>>>>>
>>>>> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []}
>>>>
>>>> I prefer this API for its compactness, thinking about offline synchronization. The smaller the docs, the better.
>>>>
>>>> Best
>>>> “Gregor”
>>>> —
>>>>
>>>>
>>>>> or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>>>>>
>>>>> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>>>>>
>>>>>
>>>>> * * *
>>>>>
>>>>>
>>>>> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>>>>>
>>>>> Thanks for reading this far and I’m looking forward to your feedback.
>>>>>
>>>>>
>>>>> Best,
>>>>> Jan “_access” Lehnardt
>>>>> —
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>>
>>>>>> Hi Everyone,
>>>>>>
>>>>>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>>>>>
>>>>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>>>>>
>>>>>> You can check out my branch here:
>>>>>>
>>>>>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>>>>>
>>>>>> It is very much work in progress, but it is far enough along to warrant discussion.
>>>>>>
>>>>>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>>>>>
>>>>>> Things I’ve left for later:
>>>>>>
>>>>>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>>>>>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>>>>>> - pretty HTTP error handling
>>>>>> - tests except for a tiny shell script 😇
>>>>>>
>>>>>> Implementation notes:
>>>>>>
>>>>>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>>>>>
>>>>>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>>>>>
>>>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>>>>>
>>>>>> * * *
>>>>>>
>>>>>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>>>>>
>>>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>>>>>
>>>>>> * * *
>>>>>>
>>>>>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>>>>>
>>>>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>>>>>
>>>>>> * * *
>>>>>>
>>>>>> Open questions:
>>>>>>
>>>>>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>>>>>
>>>>>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>>>>>
>>>>>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>>>>>
>>>>>> * * *
>>>>>>
>>>>>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>>>>>
>>>>>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>>>>>
>>>>>> Best
>>>>>> Jan
>>>>>> —
>>>>>
>>>>> --
>>>>> Professional Support for Apache CouchDB:
>>>>> https://neighbourhood.ie/couchdb-support/
>>>>>
>>>>
>>>> --
>>>> Professional Support for Apache CouchDB:
>>>> https://neighbourhood.ie/couchdb-support/
>>>>
>>>
>>> --
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/ <https://neighbourhood.ie/couchdb-support/>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
Thanks for your initial comments.
> On 3. Apr 2019, at 23:07, Adam Kocoloski <ko...@apache.org> wrote:
>
> I’m also in favor of dropping Scenario 3.
>
> One topic we may have discussed in the past but I wanted to close out here: in the relational database world it’s not uncommon to use materialized views as an access control mechanism to selectively expose contents of a table to clients who cannot access the table directly. Does the current thinking on _access for views support that use case? Can we build a view using a set of roles inherited from the user who created the design doc, but then turn around and set the _access on the view itself to a less-restrictive set?
3 minutes thinking it over didn’t reveal any particular problems with this feature, aside from include_docs not working as expected, which might be an okay trade-off for now. But could be included later.
> On the _revs_diff topic — I’m not all that concerned about users trying to guess revision IDs that exist on the server, and then reverse-engineer the contents of the existing revisions. Maybe I ought to be.
I’m not particularly worried, but it is at least a theoretical situation where our user’s can be caught with their pants down when they didn’t expect it. All I want to make sure is to document this properly. C.f. git where if you get access to a repo, you get the whole history, not just the state from where you started having access.
Best
Jan
—
>
> On a somewhat-related note, I have had conversations before with folks who are keen to adopt these sorts of fine-grained access control systems who said they actually prefer to have a 403 Forbidden response list the set of privileges that would be sufficient to access the resource. I found this surprising, but I guess it comes down to a user needing to figure out what kind of security exception to apply for in order to make progress with some data analysis. I think this is a topic on which we could make a fairly late-binding decision — or even have it as a configurable option.
>
> I could definitely see the base Scenario 1 (single _access labels) landing ahead of the more-complex sharing models.
>
> I haven’t had a chance to take a deep look at the code but the design seems good and thoughtful, and I definitely like the focus on the use cases.
>
> Adam
>
>> On Mar 14, 2019, at 11:21 AM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> My replies now inline.
>>
>>> On 14. Mar 2019, at 16:13, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> I received some notes privately from Gregor Martynus, which I’m reproducing here in email thread form. This email is all Gregor’s notes, my next email is my replies to them.
>>>
>>>> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> Hey all,
>>>>
>>>> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>>>>
>>>> My guiding principles so far are:
>>>>
>>>> 1. Make the API intuitive, things should work like they look like they should work like.
>>>> 2. The default should never be that a resources is accidentally left accessible to the public.
>>>> 3. This should work as a natural extension to the existing security features*.
>>>>
>>>> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>>>>
>>>>
>>>> ## Database Creation and Default Behaviours
>>>>
>>>> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>>>>
>>>> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>>>>
>>>> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>>>>
>>>> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>>>>
>>>> * * *
>>>>
>>>>
>>>> ## Document Creation and Access Control
>>>>
>>>> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>>>>
>>>> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>>>>
>>>> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
>>>
>>> I prefer being explicit.
>>>
>>>
>>>>
>>>> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>>>>
>>>> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well.
>>>
>>> We could also provide tooling for migrations?
>>
>> I’d love tooling, but we’d have to make sure we can do it correctly for a big number of use-cases. For the acceptance of this change, I’d make “documenting a migration path for db-per-user setups” a MUST have, and any code that helps with that a nice to have.
>>
>>>
>>>
>>>> We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>>>>
>>>> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>>>>
>>>> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>>>>
>>>> * * *
>>>>
>>>>
>>>> ## _revs_diff
>>>>
>>>> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>>>>
>>>> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>>>>
>>>> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>>>>
>>>> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>>>>
>>>> * * *
>>>>
>>>>
>>>> ## Partitioned Databases
>>>>
>>>> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>>>>
>>>> * * *
>>>>
>>>>
>>>> ## Scenarios that _access should help with.
>>>>
>>>> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>>>>
>>>>
>>>> ### Scenario 1: db-per-user
>>>>
>>>> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>>>>
>>>> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>>>>
>>>> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>>>>
>>>> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>>>>
>>>> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>>>>
>>>>
>>>> ### Scenario 2: db-per-user + Sharing
>>>
>>> One scenario we should address is how stopping to share would work when documents are continuously replicated, e.g. to a client for offline usage. My understanding is that for the person who’s access to documents got revoked does not get _changes update telling them that their access got removed, it would be up to the application developer to implement some kind of "notification" meta documents. Unless you have a better idea?
>>
>> Since we now have a purge API as well, we could treat an un-share as a purge for clients, and they can decide what to do with it.
>>
>> Alternatively, we need to make breaking changes to _changes feed, maybe we can hide that behind an opt-in flag, like “/db/_changes?access=true”, and then we can send new rows like:
>>
>> {seq: XYZ, id: abc, rev:4-YYY, _revoked: true} or somesuch.
>>
>>
>>>
>>>>
>>>> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>>>>
>>>> #### 2.1: The Todo List
>>>>
>>>> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>>>>
>>>> #### 2.2: The Chat/Forum/Newsgroup
>>>>
>>>> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>>>>
>>>> #### 2.3: The Corporate Hierarchy
>>>>
>>>> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>>>>
>>>>
>>>> ### 3: Multiple Apps
>>>>
>>>> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>>>>
>>>> I’ll be referring to these scenarios down the line.
>>>>
>>>> * * *
>>>>
>>>>
>>>> ## Design Docs
>>>>
>>>> ### Admin
>>>>
>>>> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>>>>
>>>> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>>>>
>>>>
>>>> ### User
>>>>
>>>> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>>>>
>>>> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>>>>
>>>> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>>>>
>>>> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>>>>
>>>> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs,
>>>
>>> I can’t think of a db-per-user scenario where each user DB would have a different validate_doc_update method? It would be the same method with access to the user context, the DBs security setting and the document, so it would act differently for different users, but using the same code.
>>
>> They wouldn’t be different, but if we were do replicate 1000 db-per-user design docs into a single database, as per today’s semantics, we’d have to run 1000 VDUs on each doc update.
>>
>>>
>>>> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>>>>
>>>> We could say:
>>>>
>>>> a) we don’t support scenario 3.
>>>
>>> +1, I think it would make our lives easier in general if we don’t recommend to share the same CouchDB for multiple apps. At least I don’t see a reason to do that at this point.
>>
>> I think I like this best, too, but I’d like to hear from others as well.
>>
>>
>> Best
>> Jan
>> —
>>>
>>>> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
>>>> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>>>>
>>>> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>>>>
>>>> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>>>>
>>>>
>>>> ## Read and Write Access
>>>>
>>>> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>>>>
>>>> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>>>>
>>>> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
>>>
>>> Yes!
>>>
>>>>
>>>> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []}
>>>
>>> I prefer this API for its compactness, thinking about offline synchronization. The smaller the docs, the better.
>>>
>>> Best
>>> “Gregor”
>>> —
>>>
>>>
>>>> or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>>>>
>>>> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>>>>
>>>>
>>>> * * *
>>>>
>>>>
>>>> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>>>>
>>>> Thanks for reading this far and I’m looking forward to your feedback.
>>>>
>>>>
>>>> Best,
>>>> Jan “_access” Lehnardt
>>>> —
>>>>
>>>>
>>>>
>>>>
>>>>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>>>>
>>>>> Hi Everyone,
>>>>>
>>>>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>>>>
>>>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>>>>
>>>>> You can check out my branch here:
>>>>>
>>>>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>>>>
>>>>> It is very much work in progress, but it is far enough along to warrant discussion.
>>>>>
>>>>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>>>>
>>>>> Things I’ve left for later:
>>>>>
>>>>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>>>>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>>>>> - pretty HTTP error handling
>>>>> - tests except for a tiny shell script 😇
>>>>>
>>>>> Implementation notes:
>>>>>
>>>>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>>>>
>>>>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>>>>
>>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>>>>
>>>>> * * *
>>>>>
>>>>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>>>>
>>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>>>>
>>>>> * * *
>>>>>
>>>>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>>>>
>>>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>>>>
>>>>> * * *
>>>>>
>>>>> Open questions:
>>>>>
>>>>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>>>>
>>>>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>>>>
>>>>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>>>>
>>>>> * * *
>>>>>
>>>>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>>>>
>>>>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>>>>
>>>>> Best
>>>>> Jan
>>>>> —
>>>>
>>>> --
>>>> Professional Support for Apache CouchDB:
>>>> https://neighbourhood.ie/couchdb-support/
>>>>
>>>
>>> --
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/
>>>
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/ <https://neighbourhood.ie/couchdb-support/>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
> On 4. Apr 2019, at 00:25, Robert Samuel Newson <rn...@apache.org> wrote:
>
> Hi,
>
> It’s sounds like we require a separate flag as to whether any given username or role may appear in such an informative 403 message. That is, wherever we sa that X is granted access, we have an optional flag on X to say if we can disclose its right of access to others, defaulting to false if not present.
If we make this a per-db setting, it shouldn’t be hard to add. If folks need to have this work on a per-doc basis, I’m happy to argue that that should be done outside of CouchDB :)
Best
Jan
—
>
> B.
>
>> On 3 Apr 2019, at 23:21, Adam Kocoloski <ko...@apache.org> wrote:
>>
>> Totally agree it’s information leakage - that’s why I found it surprising that this was their desired mode of operation. It works when there’s a relatively small set of labels that get applied to data, and the labels themselves are not all that confidential.
>>
>> Adam
>>
>>> On Apr 3, 2019, at 5:53 PM, Joan Touzet <wo...@apache.org> wrote:
>>>
>>> One parenthetical...
>>>
>>>> From: "Adam Kocoloski" <ko...@apache.org>
>>>>
>>>> On a somewhat-related note, I have had conversations before with
>>>> folks who are keen to adopt these sorts of fine-grained access
>>>> control systems who said they actually prefer to have a 403
>>>> Forbidden response list the set of privileges that would be
>>>> sufficient to access the resource. I found this surprising, but I
>>>> guess it comes down to a user needing to figure out what kind of
>>>> security exception to apply for in order to make progress with some
>>>> data analysis. I think this is a topic on which we could make a
>>>> fairly late-binding decision — or even have it as a configurable
>>>> option.
>>>
>>> Anyone who's ever had to deal with Amazon's AWS IAM configuration
>>> certainly can appreciate this need. I'm +1 on the idea, assuming it's
>>> not hard to implement...but...
>>>
>>> The problem is that it can be a data leak. In Jan's initial gist, he
>>> shows the _access field being populated by usernames only (Scenario 1).
>>> The only possible exception here is to get your username added to the
>>> _access field on that document.
>>>
>>> If we do this via roles, then you could be leaking role name
>>> definitions via this response. Not sure we care, but having a full
>>> list of roles that could possibly provide that permission is certainly
>>> a hole.
>>>
>>> If we do this via _capability_, then you're looking at a set of
>>> permissions such as reader, writer, deleter, and that specific
>>> permission could be returned:
>>>
>>> {"needed":"writer", "obtained":"reader"}
>>>
>>> That'd work, but it's different from what Jan has proposed to date, I
>>> believe, especially in distinguishing between read, write, and delete.
>>>
>>> -Joan
>>
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Robert Samuel Newson <rn...@apache.org>.
Hi,
It’s sounds like we require a separate flag as to whether any given username or role may appear in such an informative 403 message. That is, wherever we sa that X is granted access, we have an optional flag on X to say if we can disclose its right of access to others, defaulting to false if not present.
B.
> On 3 Apr 2019, at 23:21, Adam Kocoloski <ko...@apache.org> wrote:
>
> Totally agree it’s information leakage - that’s why I found it surprising that this was their desired mode of operation. It works when there’s a relatively small set of labels that get applied to data, and the labels themselves are not all that confidential.
>
> Adam
>
>> On Apr 3, 2019, at 5:53 PM, Joan Touzet <wo...@apache.org> wrote:
>>
>> One parenthetical...
>>
>>> From: "Adam Kocoloski" <ko...@apache.org>
>>>
>>> On a somewhat-related note, I have had conversations before with
>>> folks who are keen to adopt these sorts of fine-grained access
>>> control systems who said they actually prefer to have a 403
>>> Forbidden response list the set of privileges that would be
>>> sufficient to access the resource. I found this surprising, but I
>>> guess it comes down to a user needing to figure out what kind of
>>> security exception to apply for in order to make progress with some
>>> data analysis. I think this is a topic on which we could make a
>>> fairly late-binding decision — or even have it as a configurable
>>> option.
>>
>> Anyone who's ever had to deal with Amazon's AWS IAM configuration
>> certainly can appreciate this need. I'm +1 on the idea, assuming it's
>> not hard to implement...but...
>>
>> The problem is that it can be a data leak. In Jan's initial gist, he
>> shows the _access field being populated by usernames only (Scenario 1).
>> The only possible exception here is to get your username added to the
>> _access field on that document.
>>
>> If we do this via roles, then you could be leaking role name
>> definitions via this response. Not sure we care, but having a full
>> list of roles that could possibly provide that permission is certainly
>> a hole.
>>
>> If we do this via _capability_, then you're looking at a set of
>> permissions such as reader, writer, deleter, and that specific
>> permission could be returned:
>>
>> {"needed":"writer", "obtained":"reader"}
>>
>> That'd work, but it's different from what Jan has proposed to date, I
>> believe, especially in distinguishing between read, write, and delete.
>>
>> -Joan
>
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
Totally agree it’s information leakage - that’s why I found it surprising that this was their desired mode of operation. It works when there’s a relatively small set of labels that get applied to data, and the labels themselves are not all that confidential.
Adam
> On Apr 3, 2019, at 5:53 PM, Joan Touzet <wo...@apache.org> wrote:
>
> One parenthetical...
>
>> From: "Adam Kocoloski" <ko...@apache.org>
>>
>> On a somewhat-related note, I have had conversations before with
>> folks who are keen to adopt these sorts of fine-grained access
>> control systems who said they actually prefer to have a 403
>> Forbidden response list the set of privileges that would be
>> sufficient to access the resource. I found this surprising, but I
>> guess it comes down to a user needing to figure out what kind of
>> security exception to apply for in order to make progress with some
>> data analysis. I think this is a topic on which we could make a
>> fairly late-binding decision — or even have it as a configurable
>> option.
>
> Anyone who's ever had to deal with Amazon's AWS IAM configuration
> certainly can appreciate this need. I'm +1 on the idea, assuming it's
> not hard to implement...but...
>
> The problem is that it can be a data leak. In Jan's initial gist, he
> shows the _access field being populated by usernames only (Scenario 1).
> The only possible exception here is to get your username added to the
> _access field on that document.
>
> If we do this via roles, then you could be leaking role name
> definitions via this response. Not sure we care, but having a full
> list of roles that could possibly provide that permission is certainly
> a hole.
>
> If we do this via _capability_, then you're looking at a set of
> permissions such as reader, writer, deleter, and that specific
> permission could be returned:
>
> {"needed":"writer", "obtained":"reader"}
>
> That'd work, but it's different from what Jan has proposed to date, I
> believe, especially in distinguishing between read, write, and delete.
>
> -Joan
Re: [DISCUSS] Per-doc access control
Posted by Joan Touzet <wo...@apache.org>.
One parenthetical...
> From: "Adam Kocoloski" <ko...@apache.org>
>
> On a somewhat-related note, I have had conversations before with
> folks who are keen to adopt these sorts of fine-grained access
> control systems who said they actually prefer to have a 403
> Forbidden response list the set of privileges that would be
> sufficient to access the resource. I found this surprising, but I
> guess it comes down to a user needing to figure out what kind of
> security exception to apply for in order to make progress with some
> data analysis. I think this is a topic on which we could make a
> fairly late-binding decision — or even have it as a configurable
> option.
Anyone who's ever had to deal with Amazon's AWS IAM configuration
certainly can appreciate this need. I'm +1 on the idea, assuming it's
not hard to implement...but...
The problem is that it can be a data leak. In Jan's initial gist, he
shows the _access field being populated by usernames only (Scenario 1).
The only possible exception here is to get your username added to the
_access field on that document.
If we do this via roles, then you could be leaking role name
definitions via this response. Not sure we care, but having a full
list of roles that could possibly provide that permission is certainly
a hole.
If we do this via _capability_, then you're looking at a set of
permissions such as reader, writer, deleter, and that specific
permission could be returned:
{"needed":"writer", "obtained":"reader"}
That'd work, but it's different from what Jan has proposed to date, I
believe, especially in distinguishing between read, write, and delete.
-Joan
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
I’m also in favor of dropping Scenario 3.
One topic we may have discussed in the past but I wanted to close out here: in the relational database world it’s not uncommon to use materialized views as an access control mechanism to selectively expose contents of a table to clients who cannot access the table directly. Does the current thinking on _access for views support that use case? Can we build a view using a set of roles inherited from the user who created the design doc, but then turn around and set the _access on the view itself to a less-restrictive set?
On the _revs_diff topic — I’m not all that concerned about users trying to guess revision IDs that exist on the server, and then reverse-engineer the contents of the existing revisions. Maybe I ought to be.
On a somewhat-related note, I have had conversations before with folks who are keen to adopt these sorts of fine-grained access control systems who said they actually prefer to have a 403 Forbidden response list the set of privileges that would be sufficient to access the resource. I found this surprising, but I guess it comes down to a user needing to figure out what kind of security exception to apply for in order to make progress with some data analysis. I think this is a topic on which we could make a fairly late-binding decision — or even have it as a configurable option.
I could definitely see the base Scenario 1 (single _access labels) landing ahead of the more-complex sharing models.
I haven’t had a chance to take a deep look at the code but the design seems good and thoughtful, and I definitely like the focus on the use cases.
Adam
> On Mar 14, 2019, at 11:21 AM, Jan Lehnardt <ja...@apache.org> wrote:
>
> My replies now inline.
>
>> On 14. Mar 2019, at 16:13, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> I received some notes privately from Gregor Martynus, which I’m reproducing here in email thread form. This email is all Gregor’s notes, my next email is my replies to them.
>>
>>> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> Hey all,
>>>
>>> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>>>
>>> My guiding principles so far are:
>>>
>>> 1. Make the API intuitive, things should work like they look like they should work like.
>>> 2. The default should never be that a resources is accidentally left accessible to the public.
>>> 3. This should work as a natural extension to the existing security features*.
>>>
>>> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>>>
>>>
>>> ## Database Creation and Default Behaviours
>>>
>>> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>>>
>>> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>>>
>>> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>>>
>>> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>>>
>>> * * *
>>>
>>>
>>> ## Document Creation and Access Control
>>>
>>> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>>>
>>> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>>>
>>> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
>>
>> I prefer being explicit.
>>
>>
>>>
>>> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>>>
>>> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well.
>>
>> We could also provide tooling for migrations?
>
> I’d love tooling, but we’d have to make sure we can do it correctly for a big number of use-cases. For the acceptance of this change, I’d make “documenting a migration path for db-per-user setups” a MUST have, and any code that helps with that a nice to have.
>
>>
>>
>>> We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>>>
>>> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>>>
>>> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>>>
>>> * * *
>>>
>>>
>>> ## _revs_diff
>>>
>>> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>>>
>>> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>>>
>>> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>>>
>>> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>>>
>>> * * *
>>>
>>>
>>> ## Partitioned Databases
>>>
>>> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>>>
>>> * * *
>>>
>>>
>>> ## Scenarios that _access should help with.
>>>
>>> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>>>
>>>
>>> ### Scenario 1: db-per-user
>>>
>>> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>>>
>>> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>>>
>>> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>>>
>>> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>>>
>>> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>>>
>>>
>>> ### Scenario 2: db-per-user + Sharing
>>
>> One scenario we should address is how stopping to share would work when documents are continuously replicated, e.g. to a client for offline usage. My understanding is that for the person who’s access to documents got revoked does not get _changes update telling them that their access got removed, it would be up to the application developer to implement some kind of "notification" meta documents. Unless you have a better idea?
>
> Since we now have a purge API as well, we could treat an un-share as a purge for clients, and they can decide what to do with it.
>
> Alternatively, we need to make breaking changes to _changes feed, maybe we can hide that behind an opt-in flag, like “/db/_changes?access=true”, and then we can send new rows like:
>
> {seq: XYZ, id: abc, rev:4-YYY, _revoked: true} or somesuch.
>
>
>>
>>>
>>> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>>>
>>> #### 2.1: The Todo List
>>>
>>> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>>>
>>> #### 2.2: The Chat/Forum/Newsgroup
>>>
>>> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>>>
>>> #### 2.3: The Corporate Hierarchy
>>>
>>> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>>>
>>>
>>> ### 3: Multiple Apps
>>>
>>> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>>>
>>> I’ll be referring to these scenarios down the line.
>>>
>>> * * *
>>>
>>>
>>> ## Design Docs
>>>
>>> ### Admin
>>>
>>> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>>>
>>> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>>>
>>>
>>> ### User
>>>
>>> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>>>
>>> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>>>
>>> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>>>
>>> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>>>
>>> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs,
>>
>> I can’t think of a db-per-user scenario where each user DB would have a different validate_doc_update method? It would be the same method with access to the user context, the DBs security setting and the document, so it would act differently for different users, but using the same code.
>
> They wouldn’t be different, but if we were do replicate 1000 db-per-user design docs into a single database, as per today’s semantics, we’d have to run 1000 VDUs on each doc update.
>
>>
>>> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>>>
>>> We could say:
>>>
>>> a) we don’t support scenario 3.
>>
>> +1, I think it would make our lives easier in general if we don’t recommend to share the same CouchDB for multiple apps. At least I don’t see a reason to do that at this point.
>
> I think I like this best, too, but I’d like to hear from others as well.
>
>
> Best
> Jan
> —
>>
>>> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
>>> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>>>
>>> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>>>
>>> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>>>
>>>
>>> ## Read and Write Access
>>>
>>> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>>>
>>> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>>>
>>> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
>>
>> Yes!
>>
>>>
>>> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []}
>>
>> I prefer this API for its compactness, thinking about offline synchronization. The smaller the docs, the better.
>>
>> Best
>> “Gregor”
>> —
>>
>>
>>> or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>>>
>>> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>>>
>>>
>>> * * *
>>>
>>>
>>> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>>>
>>> Thanks for reading this far and I’m looking forward to your feedback.
>>>
>>>
>>> Best,
>>> Jan “_access” Lehnardt
>>> —
>>>
>>>
>>>
>>>
>>>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> Hi Everyone,
>>>>
>>>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>>>
>>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>>>
>>>> You can check out my branch here:
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>>>
>>>> It is very much work in progress, but it is far enough along to warrant discussion.
>>>>
>>>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>>>
>>>> Things I’ve left for later:
>>>>
>>>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>>>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>>>> - pretty HTTP error handling
>>>> - tests except for a tiny shell script 😇
>>>>
>>>> Implementation notes:
>>>>
>>>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>>>
>>>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>>>
>>>> * * *
>>>>
>>>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>>>
>>>> * * *
>>>>
>>>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>>>
>>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>>>
>>>> * * *
>>>>
>>>> Open questions:
>>>>
>>>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>>>
>>>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>>>
>>>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>>>
>>>> * * *
>>>>
>>>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>>>
>>>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>>>
>>>> Best
>>>> Jan
>>>> —
>>>
>>> --
>>> Professional Support for Apache CouchDB:
>>> https://neighbourhood.ie/couchdb-support/
>>>
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/ <https://neighbourhood.ie/couchdb-support/>
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
My replies now inline.
> On 14. Mar 2019, at 16:13, Jan Lehnardt <ja...@apache.org> wrote:
>
> I received some notes privately from Gregor Martynus, which I’m reproducing here in email thread form. This email is all Gregor’s notes, my next email is my replies to them.
>
>> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Hey all,
>>
>> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>>
>> My guiding principles so far are:
>>
>> 1. Make the API intuitive, things should work like they look like they should work like.
>> 2. The default should never be that a resources is accidentally left accessible to the public.
>> 3. This should work as a natural extension to the existing security features*.
>>
>> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>>
>>
>> ## Database Creation and Default Behaviours
>>
>> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>>
>> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>>
>> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>>
>> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>>
>> * * *
>>
>>
>> ## Document Creation and Access Control
>>
>> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>>
>> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>>
>> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
>
> I prefer being explicit.
>
>
>>
>> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>>
>> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well.
>
> We could also provide tooling for migrations?
I’d love tooling, but we’d have to make sure we can do it correctly for a big number of use-cases. For the acceptance of this change, I’d make “documenting a migration path for db-per-user setups” a MUST have, and any code that helps with that a nice to have.
>
>
>> We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>>
>> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>>
>> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>>
>> * * *
>>
>>
>> ## _revs_diff
>>
>> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>>
>> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>>
>> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>>
>> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>>
>> * * *
>>
>>
>> ## Partitioned Databases
>>
>> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>>
>> * * *
>>
>>
>> ## Scenarios that _access should help with.
>>
>> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>>
>>
>> ### Scenario 1: db-per-user
>>
>> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>>
>> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>>
>> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>>
>> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>>
>> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>>
>>
>> ### Scenario 2: db-per-user + Sharing
>
> One scenario we should address is how stopping to share would work when documents are continuously replicated, e.g. to a client for offline usage. My understanding is that for the person who’s access to documents got revoked does not get _changes update telling them that their access got removed, it would be up to the application developer to implement some kind of "notification" meta documents. Unless you have a better idea?
Since we now have a purge API as well, we could treat an un-share as a purge for clients, and they can decide what to do with it.
Alternatively, we need to make breaking changes to _changes feed, maybe we can hide that behind an opt-in flag, like “/db/_changes?access=true”, and then we can send new rows like:
{seq: XYZ, id: abc, rev:4-YYY, _revoked: true} or somesuch.
>
>>
>> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>>
>> #### 2.1: The Todo List
>>
>> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>>
>> #### 2.2: The Chat/Forum/Newsgroup
>>
>> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>>
>> #### 2.3: The Corporate Hierarchy
>>
>> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>>
>>
>> ### 3: Multiple Apps
>>
>> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>>
>> I’ll be referring to these scenarios down the line.
>>
>> * * *
>>
>>
>> ## Design Docs
>>
>> ### Admin
>>
>> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>>
>> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>>
>>
>> ### User
>>
>> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>>
>> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>>
>> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>>
>> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>>
>> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs,
>
> I can’t think of a db-per-user scenario where each user DB would have a different validate_doc_update method? It would be the same method with access to the user context, the DBs security setting and the document, so it would act differently for different users, but using the same code.
They wouldn’t be different, but if we were do replicate 1000 db-per-user design docs into a single database, as per today’s semantics, we’d have to run 1000 VDUs on each doc update.
>
>> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>>
>> We could say:
>>
>> a) we don’t support scenario 3.
>
> +1, I think it would make our lives easier in general if we don’t recommend to share the same CouchDB for multiple apps. At least I don’t see a reason to do that at this point.
I think I like this best, too, but I’d like to hear from others as well.
Best
Jan
—
>
>> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
>> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>>
>> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>>
>> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>>
>>
>> ## Read and Write Access
>>
>> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>>
>> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>>
>> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
>
> Yes!
>
>>
>> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []}
>
> I prefer this API for its compactness, thinking about offline synchronization. The smaller the docs, the better.
>
> Best
> “Gregor”
> —
>
>
>> or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>>
>> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>>
>>
>> * * *
>>
>>
>> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>>
>> Thanks for reading this far and I’m looking forward to your feedback.
>>
>>
>> Best,
>> Jan “_access” Lehnardt
>> —
>>
>>
>>
>>
>>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> Hi Everyone,
>>>
>>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>>
>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>>
>>> You can check out my branch here:
>>>
>>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>>
>>> It is very much work in progress, but it is far enough along to warrant discussion.
>>>
>>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>>
>>> Things I’ve left for later:
>>>
>>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>>> - pretty HTTP error handling
>>> - tests except for a tiny shell script 😇
>>>
>>> Implementation notes:
>>>
>>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>>
>>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>>
>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>>
>>> * * *
>>>
>>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>>
>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>>
>>> * * *
>>>
>>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>>
>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>>
>>> * * *
>>>
>>> Open questions:
>>>
>>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>>
>>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>>
>>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>>
>>> * * *
>>>
>>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>>
>>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>>
>>> Best
>>> Jan
>>> —
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
I received some notes privately from Gregor Martynus, which I’m reproducing here in email thread form. This email is all Gregor’s notes, my next email is my replies to them.
> On 10. Mar 2019, at 15:51, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hey all,
>
> after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
>
> My guiding principles so far are:
>
> 1. Make the API intuitive, things should work like they look like they should work like.
> 2. The default should never be that a resources is accidentally left accessible to the public.
> 3. This should work as a natural extension to the existing security features*.
>
> * I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
>
>
> ## Database Creation and Default Behaviours
>
> Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
>
> In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
>
> To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
>
> In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
>
> * * *
>
>
> ## Document Creation and Access Control
>
> Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
>
> 1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
>
> 2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
I prefer being explicit.
>
> This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
>
> The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well.
We could also provide tooling for migrations?
> We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
>
> For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
>
> For both creations and updates, a user MUST supply at least one role they belong to or their own username.
>
> * * *
>
>
> ## _revs_diff
>
> /db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
>
> This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
>
> The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
>
> An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
>
> * * *
>
>
> ## Partitioned Databases
>
> I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
>
> * * *
>
>
> ## Scenarios that _access should help with.
>
> Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
>
>
> ### Scenario 1: db-per-user
>
> In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
>
> In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
>
> This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
>
> The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
>
> Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
>
>
> ### Scenario 2: db-per-user + Sharing
One scenario we should address is how stopping to share would work when documents are continuously replicated, e.g. to a client for offline usage. My understanding is that for the person who’s access to documents got revoked does not get _changes update telling them that their access got removed, it would be up to the application developer to implement some kind of "notification" meta documents. Unless you have a better idea?
>
> The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
>
> #### 2.1: The Todo List
>
> In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
>
> #### 2.2: The Chat/Forum/Newsgroup
>
> In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
>
> #### 2.3: The Corporate Hierarchy
>
> In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
>
>
> ### 3: Multiple Apps
>
> The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
>
> I’ll be referring to these scenarios down the line.
>
> * * *
>
>
> ## Design Docs
>
> ### Admin
>
> One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
>
> N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
>
>
> ### User
>
> In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
>
> The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
>
> The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
>
> Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
>
> More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs,
I can’t think of a db-per-user scenario where each user DB would have a different validate_doc_update method? It would be the same method with access to the user context, the DBs security setting and the document, so it would act differently for different users, but using the same code.
> how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
>
> We could say:
>
> a) we don’t support scenario 3.
+1, I think it would make our lives easier in general if we don’t recommend to share the same CouchDB for multiple apps. At least I don’t see a reason to do that at this point
> b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
> c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
>
> I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
>
> In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
>
>
> ## Read and Write Access
>
> Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
>
> Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
>
> However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
Yes!
>
> So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []}
I prefer this API for its compactness, thinking about offline synchronization. The smaller the docs, the better.
Best
“Gregor”
—
> or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
>
> The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
>
>
> * * *
>
>
> So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
>
> Thanks for reading this far and I’m looking forward to your feedback.
>
>
> Best,
> Jan “_access” Lehnardt
> —
>
>
>
>
>> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> Hi Everyone,
>>
>> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>>
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>>
>> You can check out my branch here:
>>
>> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>>
>> It is very much work in progress, but it is far enough along to warrant discussion.
>>
>> The main point of this branch is to show all the places that we would need to change to support the proposal.
>>
>> Things I’ve left for later:
>>
>> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
>> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
>> - pretty HTTP error handling
>> - tests except for a tiny shell script 😇
>>
>> Implementation notes:
>>
>> You create a database with the _access feature turned on like so: PUT /db?access=true
>>
>> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>
>> * * *
>>
>> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>
>> * * *
>>
>> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>>
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>>
>> * * *
>>
>> Open questions:
>>
>> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>>
>> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>>
>> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>>
>> * * *
>>
>> I’ll leave this here for now, I’m sure there are a few more things to consider.
>>
>> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>>
>> Best
>> Jan
>> —
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
Hey all,
after mulling this over some more, I’d like to tackle the detailed API and behaviour for this. Especially how _access work in conjunction with existing access control features.
My guiding principles so far are:
1. Make the API intuitive, things should work like they look like they should work like.
2. The default should never be that a resources is accidentally left accessible to the public.
3. This should work as a natural extension to the existing security features*.
* I’d be up for reworking the whole lot, too, but that might be a better discussion for > 4.0.
## Database Creation and Default Behaviours
Creating a database with _access features is, as mentioned before done via a flag to PUT /database?access=true
In a 3.0 world where this would land, we already agreed that databases should be admin-only by default (instead of world read/writeable today). This is a sensible default, but that leaves us with an _access enabled database that can’t be used by anyone by server or db admins. Not very useful.
To allow arbitrary users to use the db, I suggest we use the existing _security system: i.e. if a user or a group a user belongs to is mentioned in either `admins` or `members` inside of _security, they can proceed and create documents on the db. This puts a second step burden on the application developer, but it slots cleanly into the existing security mechanisms, and doesn’t require special case handling. Alternatively, we could define that _security isn’t available in _access enabled databases, but that’s something I’d like to avoid if at all possible.
In order to make it easy to specify that “everyone in _users” should be able to use the db, I suggest we add a new role `_users` that is valid inside _security, which means “everyone in /_users” (this only excludes server admins which have full access anyway).
* * *
## Document Creation and Access Control
Next, one of our non-admin users creates a doc. There are multiple options as to how we store the _access information.
1. Automatically translate the userCtx.name of a doc creation (not an update) into the first element of the _access array. E.g. user_a PUT /db/doc {"a":1} creates this doc: {"a":1,"_access":["user_a"]}. This is a little bit counter-intuitive.
2. We require that a user puts "_access":["user_a"] in themselves. This is an explicit granting of access permissions on doc creation and I think is preferable.
This leaves the edge case of docs that have no _access member: so far I thought those docs are admin-only, with maybe a db-wide option to swap the default to public access, but I think given the explicitness of 2. we can do better: require _access for all new doc creations in access-enabled databases. A user can not create a new document without an _access field that is an array that has at least one member. For public documents, we could invent a new role _public, and admin-only docs could use the existing role _admin.
The one downside to this approach is that we won’t be able to replicate existing databases into an access-enabled database without modifying all documents. This might be a worthwhile trade-off, but we should make that decision consciously and document it well. We could allow for a special case where an _admin user can create docs that have no _access field, and those docs are treated as having only the _admin role in _access. So at least we could replicate all data in, but then require a manual step to update all docs to say, migrate an existing db-per-user app, while not accidentally exposing any docs to folks that shouldn’t read them.
For the rest of cRUD, the existing document must store one of the RUD-ing user’s name or role in its _access field.
For both creations and updates, a user MUST supply at least one role they belong to or their own username.
* * *
## _revs_diff
/db/_revs_diff can answer the question of which revisions of a document do NOT exist on a replication target: http://docs.couchdb.org/en/stable/api/database/misc.html#db-revs-diff
This would allow users to specify ids and rev(s) for docs they don’t have access too (anymore), so the result schema should be expanded to handle id: unauthorized or somesuch, something the replicator needs to know what to do with, if it encounters it (say a user got removed from the _access list inbetween the replicator opening _changes and requesting the doc).
The _revs_diff implementation would have to altered to send an unauthorized token for each doc the requesting userCtx has no access to. If we can re-use some of our existing indexes, or any other performance optimisation, that’d be great. I haven’t looked at that code at all, yet.
An important side-effect of this is, once a user has been added to a doc’s _access list, they get access to “the full history of the doc”, even before they had access. Of course, in CouchDB this means only getting access to the rev ids, and not the content, but since they are content-addressable hashes, a user could brute-force themselves into revealing certain real values from earlier incarnations of the doc. I’d rather not track _access per document revision in perpetuity, so this is something we have to be very up-front about.
* * *
## Partitioned Databases
I mentioned partitioned databases in my previous mail, and I think it is something we can document that end-users can opt into, but doesn’t require any special casing on the _access proposal. That is, if users start prefixing their doc ids with a user name or id and enable both _access and partitions, then they get all the benefits of a partitioned database, and if they choose not to, they don’t, but things keep working. There are enough use-cases to warrant both behaviours.
* * *
## Scenarios that _access should help with.
Overall, we developed _access to allow users to stop using the db-per-user architecture, but once we have per-doc-access control, folks might start using this for all manner of things. We should be clear about which scenarios we support and which we don’t.
### Scenario 1: db-per-user
In this scenario, _access enabled databases, the only way to allow mutually untrusting users to store data in a part of CouchDB that only they (and admins) have access to was giving each user their own database.
In an _access enabled database, users can CRUD/_changes/_all_docs/_revs_diff their own docs knowing no other user (aside from admins) can access those docs.
This is the simplest scenario, as all we’d have to track the owner of a document and produce by-access-id/seq indexes based on that owner.
The current prototype implementation mostly reflects this stage. Not saying this is what we should ship, but it is the easiest do implement and explain.
Aside, I might be able to be persuaded to ship this as a 2.x feature, to help those folks who don’t need anything else.
### Scenario 2: db-per-user + Sharing
The second we allow per doc auth, users will want to share those docs with other users. That’s why we initially suggested the _access field be an array, so other users and groups can be specified to have access. There are multiple scenarios in this one alone:
#### 2.1: The Todo List
In this scenario, a user has a reasonable amount of ”personal data” that they want to selectively share with one or more other users.
#### 2.2: The Chat/Forum/Newsgroup
In this scenario, a user wants to share any number of documents with a reasonable number of groups. However, since we need to limit the number of groups a user belongs to (currently 10, see below for details), this might actually not be a great solution. Or folks couldn’t be in more than 10 chat groups at a time.
#### 2.3: The Corporate Hierarchy
In this scenario, users want to share any number of docs with a reasonable number of groups in a top-down/bottom-up fashion. Think CEO shares with executives, execs share with divisions, divisions report up to their one executive, etc.
### 3: Multiple Apps
The preceding scenarios all assume that a single application is responsible for everything. However, once we allow mutually distrusting users into a single database *and* make each per-user slice work (almost) like a full standalone CouchDB database, what would stop users from using this for a multi-homing feature, where different applications are used for each user in the same database?
I’ll be referring to these scenarios down the line.
* * *
## Design Docs
### Admin
One of the downsides of db-per-user is managing design docs in the face of a changing application, that is, how to distribute new design docs across 10s of 1000+s of user dbs? It’s not impossible, but tedious. In all scenarios above but scenario 3., we could simplify this significantly. Say an admin creates a design doc, and gives all users in the db access to this design doc (this could be with the _users role, or yet another new role _members, if we need it), requesting the result of a view defined in that design doc will produce an index that is powered by the requesting user’s by-access-seq index section(s).
N.B., this would require us to change a fundamental assumption when doing the association between a design doc’s definition and index: normally, there is only the `views` member that is hashed and that hash is used as the index’s filename. Because there is only by-seq to power a view, that all works. But now that we have an arbitrary set of sections on by-access-seq, any view index built will have to take a user’s name and roles into account. When a user leaves a group, or gains a group, all indexes for that user will no longer be valid and need rebuilding.
### User
In any of the scenarios above, but especially 3., there could be legitimate per-user design docs, so how should those be treated in an _access enabled database?
The significant fields in a design doc are `views`, `validate_doc_update` and `filters` (I’ll skip over the deprecated _show, _list, and _update).
The easiest to handle is a `filters`: if a user specifies a filter for a _changes request or replication that lives in a design doc they don’t have access to, they get an error, similar to if they specify a non-existent design doc, just with `unauthorized` instead of `not_found`.
Next `views` is also not very hard to imagine working: just like globally defined views for that db, the index is built for each user based on the user’s name and roles.
More troubling are `validate_doc_update` functions: One, they are already troubling in that they slow down any document updates. Two, if we now import an existing db-per-user scenario where each user has their own design docs, how should we apply validate_doc_update functions? 10s of 1000s of VDUs are impractical to apply on each doc update, let alone just the management of VDUs that are active on a database. One option would be to ignore VDUs if they are not defined globally (say with a _members role). But especially in scenario 3. this becomes problematic, but even without that specific scenario, this violates the no surprises best practice.
We could say:
a) we don’t support scenario 3.
b) we find a complicated but efficient way to apply only those VDUs that are defined in design docs the writing user has access to plus any global ones (this would be neat but rather complicated and potentially still impractical from a performance perspective for N users).
c) we could store all per-user design docs, but ignore them completely, VDUs, views and filters.
I think I currently fall on the side of not supporting scenario 3. and asking folks who migrate db-per-user to de-duplicate design docs and keep them per-app. I believe that is a good trade-off between the most common scenarios for db-per-user while keeping the implementation manageable. Globally accessible design docs would show up in a user’s changes feed and would replicate down to say a PouchDB application which might be the exclusive user of those design docs.
In practice this would mean, a document that has an _id that starts with _design/ will have to be produced by a database admin. Luckily, that’s already the case. We should just make sure that folks don’t give db-admin access to all users habitually.
## Read and Write Access
Speaking of validate_doc_update, it is used for two things: checking document schema and doc update authorisation.
Once we allow access to a document with an _access field, we need to decide what kind of access this gives to a doc: read-only or read-write (I’m not considering write-only because for anything but doc creations this is not useful as you need access to the current _rev).
However, when we look at implementing an application on top of our existing API, it is already weird that read access can be controlled globally (or with _access on a per doc level), but write access requires writing JavaScript code. I think it would be a reasonable expectation for users to expect a per-doc read/write permission granting.
So we could have all of the above, but with two extra fields: _access_read and _access_write, or _access: {read: [], write: []} or we overload user and group names: _access: [user_a:read, user_b:write] (or any permutation thereof). Overloading can cause trouble with naturally occurring characters in group names.
The former seems more explicit, but from an API perspective that’s a little more awkward: remember that we currently have an arbitrary limit of 10 members in a user’s role array, to avoid excessive fan out on cluster-internal operations. Partitioned dbs could get away with more, more easily however. If we allow the specification of access control in two lists, and one of the lists implies membership in the other, we have a total limit of 10 members across both arrays. Or we limit 5 + 5, but that seems excessive, while 10 total seems weird, but doable. Anyway, good bikeshed.
* * *
So far. I think all of the problems outlined are solvable, if with a clear definition of what use-cases we do not support with access. If you have more scenarios than the ones I outlined, please add them and we can see if they cause any additional trouble.
Thanks for reading this far and I’m looking forward to your feedback.
Best,
Jan “_access” Lehnardt
—
> On 17. Feb 2019, at 15:25, Jan Lehnardt <ja...@apache.org> wrote:
>
> Hi Everyone,
>
> I’m happy to share my work in progress attempt to implement the per-doc access control feature we discussed a good while ago:
>
> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E <https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E>
>
> You can check out my branch here:
>
> https://github.com/apache/couchdb/compare/access?expand=1 <https://github.com/apache/couchdb/compare/access?expand=1>
>
> It is very much work in progress, but it is far enough along to warrant discussion.
>
> The main point of this branch is to show all the places that we would need to change to support the proposal.
>
> Things I’ve left for later:
>
> - currently only the first element in the _access array is used. Our and/or syntax can be added later.
> - building per-access views has not been implemented yet, couch_index would have to be taught about the new per-access-id index.
> - pretty HTTP error handling
> - tests except for a tiny shell script 😇
>
> Implementation notes:
>
> You create a database with the _access feature turned on like so: PUT /db?access=true
>
> I started out with storing _access in the document body, as that would allow for a minimal change set, however, on doc updates, we try hard not to load the old doc body from the database, and forcing us to do so for EVERY doc update under _access seemed prohibitive, so I extended the #doc, #doc_info and #full_doc_info records with a new `access` attribute that is stored in both by-id and by-seq. I will need guidance on how extending these records impact multi-version cluster interop. And especially whether this is an acceptable approach.
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>
> * * *
>
> The main addition is a new native query server called couch_access_native_proc, which implements two new indexes by-access-id and by-access-seq which do what you’d expect, pass in a userCtx and retrieve the equivalent of _all_docs or _changes, but only including those docs that match the username and roles in their _access property. The existing handlers for _all_docs and _changes have been augmented to use the new indexes instead of the default ones, unless the user is an admin.
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>
> * * *
>
> The rest of the diff is concerned with making document CRUD behave as you’d expect it. See this little demonstration for what things look like:
>
> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just noticing that there might be something wonky with DELETE, but you’ll get the gist #rimshot)
>
> * * *
>
> Open questions:
>
> - The aim of this is to get as close to regular CouchDB behaviour as possible. One thing that is new however which would require all apps to be changed is that for an _access enabled database to include an _access field in their docs (docs with no _access are admin-only for now). We might want to consider on new document writes to auto-insert the authenticated user’s name as the first element in the _access array, so existing apps “just work”.
>
> - Interplay with partitioned dbs: eschewing db-per-user is already a large boon if you have a lot of users, but making those per-user requests inside an _access enabled database efficient would be doubly nice, so why not use the username from the first question above and use that as the partition key? This would work nicely for natural users with their own docs that want to share them with others later, but I can easily imagine a pipelined use of CouchDB, where a “collector” user creates all new docs, an “analyser” takes them over and hand them to a “result” user for viewing. In that case, we’d violate the high-cardinality rule of partitions (have a lot of small ones), instead all docs go through all three users. I’d be okay with treating the later scenario as a minor use-case, but for that use-case, we should be able to disable auto-partitioning on db creation.
>
> - building access view indexes for docs that have frequent _access changes, lead to many orphaned view indexes, we should look at an auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap back and forth).
>
> * * *
>
> I’ll leave this here for now, I’m sure there are a few more things to consider.
>
> I’d love to hear any and all feedback you might have. Especially if anything is unclear.
>
> Best
> Jan
> —
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Ilya Khlopotov <ii...@apache.org>.
> The application would override the _all_docs and _changes endpoints, and if
> a user has enabled access=true for that database then you could then return
> the _all_docs and _changes requests from your application. The epi http
> work is pretty fancy I think we could do some cool things around that to
> make this work well.
I would advise against using EPI for this. The main role of the EPI (and especially override of endpoints) is to allow custom builds of CouchDB with extended functionality. In this case per-doc access control feature is part of the Apache CouchDB. If we would use override it would prevent subsequent override from the custom build.
best regards,
iilyak
On 2019/02/26 10:18:45, Garren Smith <ga...@apache.org> wrote:
> Hi Jan,
>
> I've been giving this some thought and I wonder if we should take a step
> back and rethink how we do this. Instead of implementing this directly into
> the CouchDB core code, it might be better to write this as an application
> similar to Dreyfus - Cloudant's search[1]. Instead of writing this code
> directly in the core CouchDB code rather we write this as another
> application. I'm hoping then that you wouldn't have to make huge
> modifications to the CouchDB codebase which should make this easier to do.
> The application would override the _all_docs and _changes endpoints, and if
> a user has enabled access=true for that database then you could then return
> the _all_docs and _changes requests from your application. The epi http
> work is pretty fancy I think we could do some cool things around that to
> make this work well. The app would listen to the changes feeds of any
> database that has access=true and then implement the required index's for
> _all_docs and changes. I think we then would not have to create a custom
> indexer as we could build the indexes when new changes arrive.
>
> I'm also hoping that another advantage of doing this as an app that listens
> to the changes feed is that there should be minimal work to get this to
> work when we switch to fdb.
>
> This is obviously just an idea I had and I thought I would share it, not in
> an attempt to derail what you doing, but hopefully in an attempt to make
> sure we find the easiest and most effective way to get this done.
>
> Cheers
> Garren
>
>
> [1] https://github.com/cloudant-labs/dreyfus
>
> On Sun, Feb 17, 2019 at 4:25 PM Jan Lehnardt <ja...@apache.org> wrote:
>
> > Hi Everyone,
> >
> > I’m happy to share my work in progress attempt to implement the per-doc
> > access control feature we discussed a good while ago:
> >
> >
> > https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> > <
> > https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> > >
> >
> > You can check out my branch here:
> >
> > https://github.com/apache/couchdb/compare/access?expand=1 <
> > https://github.com/apache/couchdb/compare/access?expand=1>
> >
> > It is very much work in progress, but it is far enough along to warrant
> > discussion.
> >
> > The main point of this branch is to show all the places that we would need
> > to change to support the proposal.
> >
> > Things I’ve left for later:
> >
> > - currently only the first element in the _access array is used. Our
> > and/or syntax can be added later.
> > - building per-access views has not been implemented yet, couch_index
> > would have to be taught about the new per-access-id index.
> > - pretty HTTP error handling
> > - tests except for a tiny shell script 😇
> >
> > Implementation notes:
> >
> > You create a database with the _access feature turned on like so: PUT
> > /db?access=true
> >
> > I started out with storing _access in the document body, as that would
> > allow for a minimal change set, however, on doc updates, we try hard not to
> > load the old doc body from the database, and forcing us to do so for EVERY
> > doc update under _access seemed prohibitive, so I extended the #doc,
> > #doc_info and #full_doc_info records with a new `access` attribute that is
> > stored in both by-id and by-seq. I will need guidance on how extending
> > these records impact multi-version cluster interop. And especially whether
> > this is an acceptable approach.
> >
> >
> > https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
> >
> > * * *
> >
> > The main addition is a new native query server called
> > couch_access_native_proc, which implements two new indexes by-access-id and
> > by-access-seq which do what you’d expect, pass in a userCtx and retrieve
> > the equivalent of _all_docs or _changes, but only including those docs that
> > match the username and roles in their _access property. The existing
> > handlers for _all_docs and _changes have been augmented to use the new
> > indexes instead of the default ones, unless the user is an admin.
> >
> >
> > https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
> >
> > * * *
> >
> > The rest of the diff is concerned with making document CRUD behave as
> > you’d expect it. See this little demonstration for what things look like:
> >
> > https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <
> > https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just
> > noticing that there might be something wonky with DELETE, but you’ll get
> > the gist #rimshot)
> >
> > * * *
> >
> > Open questions:
> >
> > - The aim of this is to get as close to regular CouchDB behaviour as
> > possible. One thing that is new however which would require all apps to be
> > changed is that for an _access enabled database to include an _access field
> > in their docs (docs with no _access are admin-only for now). We might want
> > to consider on new document writes to auto-insert the authenticated user’s
> > name as the first element in the _access array, so existing apps “just
> > work”.
> >
> > - Interplay with partitioned dbs: eschewing db-per-user is already a large
> > boon if you have a lot of users, but making those per-user requests inside
> > an _access enabled database efficient would be doubly nice, so why not use
> > the username from the first question above and use that as the partition
> > key? This would work nicely for natural users with their own docs that want
> > to share them with others later, but I can easily imagine a pipelined use
> > of CouchDB, where a “collector” user creates all new docs, an “analyser”
> > takes them over and hand them to a “result” user for viewing. In that case,
> > we’d violate the high-cardinality rule of partitions (have a lot of small
> > ones), instead all docs go through all three users. I’d be okay with
> > treating the later scenario as a minor use-case, but for that use-case, we
> > should be able to disable auto-partitioning on db creation.
> >
> > - building access view indexes for docs that have frequent _access
> > changes, lead to many orphaned view indexes, we should look at an
> > auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap
> > back and forth).
> >
> > * * *
> >
> > I’ll leave this here for now, I’m sure there are a few more things to
> > consider.
> >
> > I’d love to hear any and all feedback you might have. Especially if
> > anything is unclear.
> >
> > Best
> > Jan
> > —
>
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
Hanks Adam and Robert for sorting this one.
Michael, the idea is to give mutually untrusting users access a as-close-to-verbatim-CouchDB API to their section of a shared database. So you get full doc CRUD, _changes, views, replication, the lot, but only for documents that you have access to. So there is no sneaking behind the back and getting all the data, if you don’t already have access to it, in which case you already have access to them :)
Best
Jan
—
> On 27. Feb 2019, at 22:55, Adam Kocoloski <ko...@apache.org> wrote:
>
>
>> On Feb 27, 2019, at 3:47 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>>
>>>
>>>
>>>> This might be what is already planned (it hasn't sounded like it to me
>>>> though).
>>>> And I definitely think changing the perspective to make "databases" a
>>>> function of the access control system and to make views based on "access
>>>> controlled collection results" instead of "databases" would be quite
>>>> powerful...
>>>>
>>>> Regards,
>>>> Mike
>>>
>>> Hi Mike, what you’ve described here is very very similar to what Jan is
>>> building.
>>>
>>> Adam
>>>
>>>
>> I read back through the links that Jan posted again; the details I was
>> looking for are probably somewhere in the sharding conversation that my
>> eyes glazed over on or somewhere in the notes of the roadmap discussion
>> which made it a bit hard for me to find just the parts related to this (I
>> most likely scrolled through it). ;-)
>>
>> Thanks for clarifying for me, and for letting me chime in!
>>
>> Mike
>
> Those details are really hard to find — I can only find them because I know exactly where to look in the minutes of a meeting that I attended well over a year ago :) Probably a good case for an RFC so we have a current pointer to the plan.
>
> Adam
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
> On Feb 27, 2019, at 3:47 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>
>>
>>
>>> This might be what is already planned (it hasn't sounded like it to me
>>> though).
>>> And I definitely think changing the perspective to make "databases" a
>>> function of the access control system and to make views based on "access
>>> controlled collection results" instead of "databases" would be quite
>>> powerful...
>>>
>>> Regards,
>>> Mike
>>
>> Hi Mike, what you’ve described here is very very similar to what Jan is
>> building.
>>
>> Adam
>>
>>
> I read back through the links that Jan posted again; the details I was
> looking for are probably somewhere in the sharding conversation that my
> eyes glazed over on or somewhere in the notes of the roadmap discussion
> which made it a bit hard for me to find just the parts related to this (I
> most likely scrolled through it). ;-)
>
> Thanks for clarifying for me, and for letting me chime in!
>
> Mike
Those details are really hard to find — I can only find them because I know exactly where to look in the minutes of a meeting that I attended well over a year ago :) Probably a good case for an RFC so we have a current pointer to the plan.
Adam
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
>
>
> > This might be what is already planned (it hasn't sounded like it to me
> > though).
> > And I definitely think changing the perspective to make "databases" a
> > function of the access control system and to make views based on "access
> > controlled collection results" instead of "databases" would be quite
> > powerful...
> >
> > Regards,
> > Mike
>
> Hi Mike, what you’ve described here is very very similar to what Jan is
> building.
>
> Adam
>
>
I read back through the links that Jan posted again; the details I was
looking for are probably somewhere in the sharding conversation that my
eyes glazed over on or somewhere in the notes of the roadmap discussion
which made it a bit hard for me to find just the parts related to this (I
most likely scrolled through it). ;-)
Thanks for clarifying for me, and for letting me chime in!
Mike
Re: [DISCUSS] Per-doc access control
Posted by Robert Newson <rn...@apache.org>.
Not to pile on here but "Once you read your available docs into your DB, you can grant yourself
write privileges to every document there." does seem to miss the mark.
All replication is doing is making a copy of data you have access to. You can modify your own copy as you please, it doesn't violate the security of the origin server. If that server allowed you to replicate those changes back, sure, but that is wholly within the origin servers control.
The notion that access controls could meaningful propagate to servers outside of the original servers control seems very much like DRM with all its doesn't-work-without-litigation problems.
--
Robert Samuel Newson
rnewson@apache.org
On Wed, 27 Feb 2019, at 20:14, Adam Kocoloski wrote:
>
> > On Feb 27, 2019, at 3:01 PM, Michael Fair <mi...@daclubhouse.net> wrote:
> >
> > On Wed, Feb 27, 2019 at 10:36 AM Adam Kocoloski <ko...@apache.org> wrote:
> >
> >> Hi Mike, just picking out this one snippet:
> >>
> >>> On Feb 27, 2019, at 12:16 PM, Michael Fair <mi...@daclubhouse.net>
> >> wrote:
> >>>
> >>> If I get a replica of a database from your server, what, if anything,
> >>> prevents me from granting myself access controls to the entire database?
> >>
> >> Replication is a client of the API like everyone else and cannot bypass
> >> the access controls on the source. You can only create a replication which
> >> has at most access to all the documents in the database that you can access
> >> yourself; i.e. a replication of a database with per-doc access controls
> >> enabled may only transfer a subset of the documents in the database.
> >>
> >
> > Right, but generally speaking READ access is very prevalent, WRITE access
> > is much more restrictive.
> >
> > Or are these "access controls" really just the binary "exposed"/"not
> > exposed" sort such that had these documents simply gone into a different
> > database in the first place, and the view indexes tracked in a "per
> > database" way then everything would work as expected?
> >
> > In fact, maybe "the same doc in multiple user centered databases" is not
> > such a bad idea/model to consider.
> >
> > Currently, I see this idea in Couch that documents belong to a particular
> > "named document collection" called a database. A view is really just
> > another kind of "named document collection".
> >
> > What if instead of a "database", there was simply the single universal
> > document store and a "database" is then more like a "view" that documents
> > became a member of just like a user's access controlled "slice" would be?
> > (i.e. the dbname becomes more like of a document access filter and less a
> > grouping and storage boundary)
> >
> > Users then have a "per user scope/database" which they always access
> > through when they connect to the server.
> > The top level list of "_databases" the user sees is actually just a list of
> > those "named collections" that the user can access. To the client side
> > APIs nothing really changes for them, what changes is how Couch internally
> > organizes itself to make "databases" a construct of the access control
> > feature and no longer a structural primitive.
> >
> > Documents now carry the information about what "named collections" they
> > belong to in the same way as what entities are authorized to access them,
> > and "databases" basically become a "view" grouping by the "_collections"
> > array field member values on each document.
> >
> >
> > Many people have requested the option to expose the same document to
> > multiple databases and then use database access rights to enforce per user
> > access and share documents between users. I've even wanted this feature
> > from time to time. If views have to be modified to hide unauthorized
> > documents anyway, this seems a perfect opportunity to address this at the
> > same time...
> >
> >
> > .........
> > In this model, a "document collection", whether it be a database, a user, a
> > user group, a role, or a role group becomes the entity that the access
> > control documents are "authorizing".
> >
> > In this model, a "view" becomes an identifiable "collection" that can be
> > treated like a database.
> > Which would make creating views on top of other views becomes something
> > much easier to define/express.
> >
> >
> > I'm envisioning that to implement successful "access control" based views,
> > each "authorized entity" would have to maintain its own view index.
> > Otherwise it's really hard for me to imagine how a reduce function can
> > cache any precomputed results because it has no way of knowing that all the
> > underlying documents used in the reduce were authorized for the accessing
> > entity... All reduce functions would have to run in real time to ensure it
> > is only using authorized documents...
> >
> > If there were a separate index for each authorized entity though (and
> > especially if these entities were able to share index buckets where they
> > had all the same document access in common), then the reduce function would
> > cache data based on each authorized entity because it'd be built up from a
> > different view index.
> >
> > The existing model where indexes are updated whenever they are accessed
> > could remain in tact because you wouldn't have to rebuild indeces for
> > entities that weren't accessing that view...
> >
> > This might be what is already planned (it hasn't sounded like it to me
> > though).
> > And I definitely think changing the perspective to make "databases" a
> > function of the access control system and to make views based on "access
> > controlled collection results" instead of "databases" would be quite
> > powerful...
> >
> > Regards,
> > Mike
>
> Hi Mike, what you’ve described here is very very similar to what Jan is
> building.
>
> Adam
>
>
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
> On Feb 27, 2019, at 3:01 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>
> On Wed, Feb 27, 2019 at 10:36 AM Adam Kocoloski <ko...@apache.org> wrote:
>
>> Hi Mike, just picking out this one snippet:
>>
>>> On Feb 27, 2019, at 12:16 PM, Michael Fair <mi...@daclubhouse.net>
>> wrote:
>>>
>>> If I get a replica of a database from your server, what, if anything,
>>> prevents me from granting myself access controls to the entire database?
>>
>> Replication is a client of the API like everyone else and cannot bypass
>> the access controls on the source. You can only create a replication which
>> has at most access to all the documents in the database that you can access
>> yourself; i.e. a replication of a database with per-doc access controls
>> enabled may only transfer a subset of the documents in the database.
>>
>
> Right, but generally speaking READ access is very prevalent, WRITE access
> is much more restrictive.
>
> Or are these "access controls" really just the binary "exposed"/"not
> exposed" sort such that had these documents simply gone into a different
> database in the first place, and the view indexes tracked in a "per
> database" way then everything would work as expected?
>
> In fact, maybe "the same doc in multiple user centered databases" is not
> such a bad idea/model to consider.
>
> Currently, I see this idea in Couch that documents belong to a particular
> "named document collection" called a database. A view is really just
> another kind of "named document collection".
>
> What if instead of a "database", there was simply the single universal
> document store and a "database" is then more like a "view" that documents
> became a member of just like a user's access controlled "slice" would be?
> (i.e. the dbname becomes more like of a document access filter and less a
> grouping and storage boundary)
>
> Users then have a "per user scope/database" which they always access
> through when they connect to the server.
> The top level list of "_databases" the user sees is actually just a list of
> those "named collections" that the user can access. To the client side
> APIs nothing really changes for them, what changes is how Couch internally
> organizes itself to make "databases" a construct of the access control
> feature and no longer a structural primitive.
>
> Documents now carry the information about what "named collections" they
> belong to in the same way as what entities are authorized to access them,
> and "databases" basically become a "view" grouping by the "_collections"
> array field member values on each document.
>
>
> Many people have requested the option to expose the same document to
> multiple databases and then use database access rights to enforce per user
> access and share documents between users. I've even wanted this feature
> from time to time. If views have to be modified to hide unauthorized
> documents anyway, this seems a perfect opportunity to address this at the
> same time...
>
>
> .........
> In this model, a "document collection", whether it be a database, a user, a
> user group, a role, or a role group becomes the entity that the access
> control documents are "authorizing".
>
> In this model, a "view" becomes an identifiable "collection" that can be
> treated like a database.
> Which would make creating views on top of other views becomes something
> much easier to define/express.
>
>
> I'm envisioning that to implement successful "access control" based views,
> each "authorized entity" would have to maintain its own view index.
> Otherwise it's really hard for me to imagine how a reduce function can
> cache any precomputed results because it has no way of knowing that all the
> underlying documents used in the reduce were authorized for the accessing
> entity... All reduce functions would have to run in real time to ensure it
> is only using authorized documents...
>
> If there were a separate index for each authorized entity though (and
> especially if these entities were able to share index buckets where they
> had all the same document access in common), then the reduce function would
> cache data based on each authorized entity because it'd be built up from a
> different view index.
>
> The existing model where indexes are updated whenever they are accessed
> could remain in tact because you wouldn't have to rebuild indeces for
> entities that weren't accessing that view...
>
> This might be what is already planned (it hasn't sounded like it to me
> though).
> And I definitely think changing the perspective to make "databases" a
> function of the access control system and to make views based on "access
> controlled collection results" instead of "databases" would be quite
> powerful...
>
> Regards,
> Mike
Hi Mike, what you’ve described here is very very similar to what Jan is building.
Adam
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
On Wed, Feb 27, 2019 at 10:36 AM Adam Kocoloski <ko...@apache.org> wrote:
> Hi Mike, just picking out this one snippet:
>
> > On Feb 27, 2019, at 12:16 PM, Michael Fair <mi...@daclubhouse.net>
> wrote:
> >
> > If I get a replica of a database from your server, what, if anything,
> > prevents me from granting myself access controls to the entire database?
>
> Replication is a client of the API like everyone else and cannot bypass
> the access controls on the source. You can only create a replication which
> has at most access to all the documents in the database that you can access
> yourself; i.e. a replication of a database with per-doc access controls
> enabled may only transfer a subset of the documents in the database.
>
Right, but generally speaking READ access is very prevalent, WRITE access
is much more restrictive.
Or are these "access controls" really just the binary "exposed"/"not
exposed" sort such that had these documents simply gone into a different
database in the first place, and the view indexes tracked in a "per
database" way then everything would work as expected?
In fact, maybe "the same doc in multiple user centered databases" is not
such a bad idea/model to consider.
Currently, I see this idea in Couch that documents belong to a particular
"named document collection" called a database. A view is really just
another kind of "named document collection".
What if instead of a "database", there was simply the single universal
document store and a "database" is then more like a "view" that documents
became a member of just like a user's access controlled "slice" would be?
(i.e. the dbname becomes more like of a document access filter and less a
grouping and storage boundary)
Users then have a "per user scope/database" which they always access
through when they connect to the server.
The top level list of "_databases" the user sees is actually just a list of
those "named collections" that the user can access. To the client side
APIs nothing really changes for them, what changes is how Couch internally
organizes itself to make "databases" a construct of the access control
feature and no longer a structural primitive.
Documents now carry the information about what "named collections" they
belong to in the same way as what entities are authorized to access them,
and "databases" basically become a "view" grouping by the "_collections"
array field member values on each document.
Many people have requested the option to expose the same document to
multiple databases and then use database access rights to enforce per user
access and share documents between users. I've even wanted this feature
from time to time. If views have to be modified to hide unauthorized
documents anyway, this seems a perfect opportunity to address this at the
same time...
.........
In this model, a "document collection", whether it be a database, a user, a
user group, a role, or a role group becomes the entity that the access
control documents are "authorizing".
In this model, a "view" becomes an identifiable "collection" that can be
treated like a database.
Which would make creating views on top of other views becomes something
much easier to define/express.
I'm envisioning that to implement successful "access control" based views,
each "authorized entity" would have to maintain its own view index.
Otherwise it's really hard for me to imagine how a reduce function can
cache any precomputed results because it has no way of knowing that all the
underlying documents used in the reduce were authorized for the accessing
entity... All reduce functions would have to run in real time to ensure it
is only using authorized documents...
If there were a separate index for each authorized entity though (and
especially if these entities were able to share index buckets where they
had all the same document access in common), then the reduce function would
cache data based on each authorized entity because it'd be built up from a
different view index.
The existing model where indexes are updated whenever they are accessed
could remain in tact because you wouldn't have to rebuild indeces for
entities that weren't accessing that view...
This might be what is already planned (it hasn't sounded like it to me
though).
And I definitely think changing the perspective to make "databases" a
function of the access control system and to make views based on "access
controlled collection results" instead of "databases" would be quite
powerful...
Regards,
Mike
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
>
> There are certainly a bunch of interesting challenges around providing
> identities that are meaningful across multiple servers in different
> domains, and I think that’s worth digging into, but I wanted to avoid
> anyone thinking that replication could trivially defeat the per-doc _access
> controls that Jan has been working on here. Cheers,
>
>
I think it depends on what you mean by "defeat".
I haven't heard enough that suggests replication can't trivially defeat
access controls yet....
That's exactly the question I'm asking.
Let's look at two specific cases:
----------------
So in one case, it's a "pull only" replica; say you are pulling from my
server to yours.
Once you read your available docs into your DB, you can grant yourself
write privileges to every document there.
My original database is obviously unchanged, but in a more p2p environment
where the idea is that the data is "daisy chaining" between replicas in a
"mesh" instead of a "star" topology; the access controls on documents in
the mesh is completely untrustworthy if those kinds of rewrites can happen.
The other case is a bi-directional replication scenario because we want
both our servers to have "some" write access to "some" documents in a
database that we share, such that we wish to replicate the full set of
documents between each other but maintain access controls. This case seems
to completely break down because you can modify any document you have read
access to and then I could easily replicate those changes back from you.
.........
In the scenarios I'm envisioning, once a document gets onto a machine, the
administrator of that machine can completely rewrite the access controls to
any document (and by extension rewrite the document itself); if those
changes are able to replicate out to other databases, and especially back
up to the original source, that's what I see poses a problem.
I suppose one fix idea is that when replicating, outbound uses read access
controls and inbound enforces write access controls; but this would have to
be at the level of tracking what entity made the modifications and not the
entity doing the replication.
I'm looking at scenarios where there is more than one user accessing the
replica because it's a multimaster distributed infoset scenario.
Perhaps you are thinking that the model is more "hub and spoke" where the
replica is only for a singular user?
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
Hi Mike, just picking out this one snippet:
> On Feb 27, 2019, at 12:16 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>
> If I get a replica of a database from your server, what, if anything,
> prevents me from granting myself access controls to the entire database?
Replication is a client of the API like everyone else and cannot bypass the access controls on the source. You can only create a replication which has at most access to all the documents in the database that you can access yourself; i.e. a replication of a database with per-doc access controls enabled may only transfer a subset of the documents in the database.
There are certainly a bunch of interesting challenges around providing identities that are meaningful across multiple servers in different domains, and I think that’s worth digging into, but I wanted to avoid anyone thinking that replication could trivially defeat the per-doc _access controls that Jan has been working on here. Cheers,
Adam
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
>
> Thanks Mike, I did understand you correctly the first time. I still
maintain that’s in the realm of authentication, not authorization, and
should be cleanly separable from the problem of implementing per-document
access controls.
>
Then I think we're saying the same thing except I'm looking/asking about
the actual intended mechanics of this particular cross domain case.
I agree access controls are an authorization question, but what the heck is
getting authorized and how do those things map to whatever gets
authenticated?
I was specifically proposing the idea that "role entities", stored inside
the database, are the only objects an authorization record could get
granted permissions to as a proposed nomenclature.
* Roles are authorized.
* Users are authenticated.
* Users are a server/system level entity.
* Roles are a database level entity.
* The valid role list replicates with the database.
* The document authorization descriptors replicate with the database.
* The users database does not replicate.
* The links between users and roles do not replicate.
And there is this big ol' security question about what data in the database
replicates to another site?
If I get a replica of a database from your server, what, if anything,
prevents me from granting myself access controls to the entire database?
If I get a replica of a database, how does that map my preexisitng
autheticated user entities onto the authorized entities of the access
control docs?
(If there's an authorized entity called 'sam' in the access controls, do I
just create a user account called 'sam' with a password I know so I can
autheticate as 'sam', and get access to Sam's doc authorizations?)
The response of "this is an authentication versus authorization issue and
we're discussing authorizations" is something I agree with; but it doesn't
seem to address my question/observation that there is a linking
relationship between the authenticated and authorized entities; and in a
cross domain replicated environment the adminstrative controls for
assigning the links and creating the respective entities are under
differing authorities/controls.
IOW, defeating Couch Access Controls looks like it will simply be pulling
the database onto your own server and granting yourself access. That doing
something more sophisticated will require a domain aware approach to
defining authenticated and authorized entities...
Is the intended design architecture a simple 'string match on the
authenticated user name' model?
IOW, is the user/authorization linking model that every user named 'sam' is
accepted/authenticated as the same 'sam' regardless of hosting server, and
therefore all users named 'sam' on every server are always authorized the
same way? I think it's obvious that while a valid way to handle it, that's
clearly suboptimal and in some cases will be completely unusable.
I agree access controls are an "authorization" question; my
questions/concerns/focus is "What conceptual entity are these access
control documents authorizing" and "How do authenticated entities get
mapped to those authorized entities"?
Specifically in a cross domain setting where the authentication systems are
under totally different administrative domains; and each domain can
effectively admin itself over the entire DB.
If you have bi-directional replication between two servers as I described
before (the shared product catalog); is there a design element here to
enforce those access controls in "cross domain safe" way?
I'm not saying there has to be, but I do think that'd be a good thing, and
I don't see an immediately obvious 'how to do it' solution...
Thanks,
Mike
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
It’s possible, but I don’t see how to scale it efficiently.
If you let a user create a view and access it directly, you need to ensure that the system which reads the documents only reads the ones that the user can access. Effectively, the design doc needs to inherit the access roles of the user who created it. An external application could enforce this although it would somehow need to be a different syntax so the regular view system doesn’t pick up the ddoc and build the view while reading all documents.
Assuming you take care of the above, the next challenge is a performance one. If you have 100 users creating their own views off of the vanilla _changes feed of a single database, the indexing process for each view is 100x more expensive than it needs to be (because each indexer scans and discards 99% of the docs due to _access restrictions). The in-database approach tries to address this by creating a _changes feed where the entries are prefixed by the _access required to read them, so an indexer can quickly skip the ranges that it knows are inaccessible.
Adam
> On Feb 27, 2019, at 6:47 AM, Garren Smith <ga...@apache.org> wrote:
>
> Hi Adam,
>
> I probably didn’t give views the most thought. I was thinking we could
> implement views by either using the changes feed or a new query engine like
> we do for mango. Does that make sense or is it not really possible?
>
>
> On Wed, Feb 27, 2019 at 2:26 AM Adam Kocoloski <ko...@apache.org> wrote:
>
>>
>>> On Feb 26, 2019, at 7:12 PM, Michael Fair <mi...@daclubhouse.net>
>> wrote:
>>>
>>> On Tue, Feb 26, 2019 at 3:38 PM Adam Kocoloski <ko...@apache.org>
>> wrote:
>>>
>>>> Mike,
>>>>
>>>> If I’m reading you correctly you’re concerned about cross-domain
>>>> authentication. A good problem and worth discussing, but I think it’s
>>>> cleanly decoupled from the per-doc access control work, which is
>> focused on
>>>> *authorization*.
>>>>
>>>>
>>>
>>> I don't think I'm talking about the same cross domain authentication you
>>> are talking about. I think you are talking about a web page from Domain
>>> (B) attempting to access Couch resource in domain (A) (Cross site
>> scripting
>>> access). That's not what I'm talking about.
>>>
>>> I'm talking about what ought to happen with the authorization control
>>> definitions when you have two Couch servers, one running in Domain (A)
>> and
>>> one running in Domain (B) with different sets of system users, such that
>>> the authorized entities in the bidirectionally replicated database don't
>>> exist in both server instances (the two distinct domains share the same
>>> document database but have disparate sets of authenticated system users).
>>>
>>> In other words the ("sam", "pete", and "joe") users on domain/machine A
>> are
>>> not the same thing as the ("mary", "betty", and "sue") users on
>>> domain/machine B; yet the replicated database between the two machines
>> has
>>> the same access control document authorization descriptors in both
>> places.
>>
>>
>> Thanks Mike, I did understand you correctly the first time. I still
>> maintain that’s in the realm of authentication, not authorization, and
>> should be cleanly separable from the problem of implementing per-document
>> access controls. Cheers,
>>
>> Adam
Re: [DISCUSS] Per-doc access control
Posted by Garren Smith <ga...@apache.org>.
Hi Adam,
I probably didn’t give views the most thought. I was thinking we could
implement views by either using the changes feed or a new query engine like
we do for mango. Does that make sense or is it not really possible?
On Wed, Feb 27, 2019 at 2:26 AM Adam Kocoloski <ko...@apache.org> wrote:
>
> > On Feb 26, 2019, at 7:12 PM, Michael Fair <mi...@daclubhouse.net>
> wrote:
> >
> > On Tue, Feb 26, 2019 at 3:38 PM Adam Kocoloski <ko...@apache.org>
> wrote:
> >
> >> Mike,
> >>
> >> If I’m reading you correctly you’re concerned about cross-domain
> >> authentication. A good problem and worth discussing, but I think it’s
> >> cleanly decoupled from the per-doc access control work, which is
> focused on
> >> *authorization*.
> >>
> >>
> >
> > I don't think I'm talking about the same cross domain authentication you
> > are talking about. I think you are talking about a web page from Domain
> > (B) attempting to access Couch resource in domain (A) (Cross site
> scripting
> > access). That's not what I'm talking about.
> >
> > I'm talking about what ought to happen with the authorization control
> > definitions when you have two Couch servers, one running in Domain (A)
> and
> > one running in Domain (B) with different sets of system users, such that
> > the authorized entities in the bidirectionally replicated database don't
> > exist in both server instances (the two distinct domains share the same
> > document database but have disparate sets of authenticated system users).
> >
> > In other words the ("sam", "pete", and "joe") users on domain/machine A
> are
> > not the same thing as the ("mary", "betty", and "sue") users on
> > domain/machine B; yet the replicated database between the two machines
> has
> > the same access control document authorization descriptors in both
> places.
>
>
> Thanks Mike, I did understand you correctly the first time. I still
> maintain that’s in the realm of authentication, not authorization, and
> should be cleanly separable from the problem of implementing per-document
> access controls. Cheers,
>
> Adam
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
> On Feb 26, 2019, at 7:12 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>
> On Tue, Feb 26, 2019 at 3:38 PM Adam Kocoloski <ko...@apache.org> wrote:
>
>> Mike,
>>
>> If I’m reading you correctly you’re concerned about cross-domain
>> authentication. A good problem and worth discussing, but I think it’s
>> cleanly decoupled from the per-doc access control work, which is focused on
>> *authorization*.
>>
>>
>
> I don't think I'm talking about the same cross domain authentication you
> are talking about. I think you are talking about a web page from Domain
> (B) attempting to access Couch resource in domain (A) (Cross site scripting
> access). That's not what I'm talking about.
>
> I'm talking about what ought to happen with the authorization control
> definitions when you have two Couch servers, one running in Domain (A) and
> one running in Domain (B) with different sets of system users, such that
> the authorized entities in the bidirectionally replicated database don't
> exist in both server instances (the two distinct domains share the same
> document database but have disparate sets of authenticated system users).
>
> In other words the ("sam", "pete", and "joe") users on domain/machine A are
> not the same thing as the ("mary", "betty", and "sue") users on
> domain/machine B; yet the replicated database between the two machines has
> the same access control document authorization descriptors in both places.
Thanks Mike, I did understand you correctly the first time. I still maintain that’s in the realm of authentication, not authorization, and should be cleanly separable from the problem of implementing per-document access controls. Cheers,
Adam
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
On Tue, Feb 26, 2019 at 3:38 PM Adam Kocoloski <ko...@apache.org> wrote:
> Mike,
>
> If I’m reading you correctly you’re concerned about cross-domain
> authentication. A good problem and worth discussing, but I think it’s
> cleanly decoupled from the per-doc access control work, which is focused on
> *authorization*.
>
>
I don't think I'm talking about the same cross domain authentication you
are talking about. I think you are talking about a web page from Domain
(B) attempting to access Couch resource in domain (A) (Cross site scripting
access). That's not what I'm talking about.
I'm talking about what ought to happen with the authorization control
definitions when you have two Couch servers, one running in Domain (A) and
one running in Domain (B) with different sets of system users, such that
the authorized entities in the bidirectionally replicated database don't
exist in both server instances (the two distinct domains share the same
document database but have disparate sets of authenticated system users).
In other words the ("sam", "pete", and "joe") users on domain/machine A are
not the same thing as the ("mary", "betty", and "sue") users on
domain/machine B; yet the replicated database between the two machines has
the same access control document authorization descriptors in both places.
Re: [DISCUSS] Per-doc access control
Posted by Adam Kocoloski <ko...@apache.org>.
Mike,
If I’m reading you correctly you’re concerned about cross-domain authentication. A good problem and worth discussing, but I think it’s cleanly decoupled from the per-doc access control work, which is focused on *authorization*.
Garren,
A lot of the complexity in per-doc access controls comes down to ensuring the view system does not inadvertently allow the exposure of information that it should not expose. Did you have views in scope when you were thinking about how this would work as an external application?
Adam
> On Feb 26, 2019, at 1:36 PM, Michael Fair <mi...@daclubhouse.net> wrote:
>
> One thing I've always been concerned about when it comes to "user based"
> document access is how it interacts with cross domain replication, and
> mobile replication (or "user sliced" replication).
>
> I believe this requires some kind of concept to a "Decentralized ID", or
> mapping user ids (roles) per database to system users (users) because the
> idea of a "user" and who the set of users are across administrative domains
> isn't likely to be the same set.
>
> Picture the same database replicating with each other between two
> adminstrative domains; like say your domain and my domain ran two separate
> Couch instances and we shared a common database. For example, a "product
> catalog", with per user access controls turned on so your organization can
> edit products and their descriptions and prices and mine could create
> purchase orders, and we both can update "tickets/issues/returns".
>
> I think its super important to figure out how this case ought to behave
> when considering the user access design. We will most likely have
> different user sets in our Couch server installations.
>
> I suggest considering that all document databases only have the concept of
> role ids, and not user ids, internal to the database and that privileges be
> granted to those roles. It's a requirement to create both a database role
> and a system user and link the two to get access to the database; the
> default role is called "public" which by default has read/write access to
> all documents and users are automatically linked/mapped to it.
>
> It's also fine to name a database role the same as a system userid (e.g.
> the 'mfair' role); but that the database not have a concept of a "user"
> (authentication) only the "role" (authorization). The system daemon
> handles autheticating users and there's a mapping of system users to
> database roles (like the 'mfair' system user could map to the 'mfair'
> database role). The roles (and access privileges) would replicate with the
> database but the system users would not.
>
> This isn't a completely baked thought yet, but I think it goes in the right
> direction.
>
> Each independent server in a separate administrative domain would have its
> own set of users, each server then has to map those system users to
> database roles. This should only have to be done once.
>
> Couch servers could also choose to replicate their user databases too.
> Multiple Couch servers that replicate their user databases with each other
> is what I'm calling an "administrative domain" or simply "domain" for short.
>
> A user can also be mapped to (i.e. hold) multiple roles in the same
> database simultaneously.
>
> This model is very close to how MS SQL Server and other SQL databases
> handles users.
>
> ..........
> Other ideas include:
> - Disclaim that replication with other domains simply breaks user access
> controls. It's too complicated so its not supported.
>
> - Replication can only be done within the context of a role that has
> read/write access to the database's documents (a new replication
> parameter). Authenticating this login for automated replication might be
> tricky (remote user credentials stored in the replication database?)...
>
> - putting all the users inside the database itself so that those ids
> replicate in addition to the contents but then every server has to be
> entrusted to authenticate every user in every database it shares with other
> domains amd the set of users becomes the superset of all users across all
> participating domains. Bring able to reset a user's password in another
> domain because you have access to manipulating the database's users seems
> "wrong"...
>
> - Using a decentralized p2p identity scheme like pki and using Couch itself
> as a distributed public key store. This has the advantage that docs can be
> encrypted and decryption secrets protected by keypairs so remote databases
> can't automatically read contents they shouldn't... It's obviously more
> complicated than simply trusting the remote administrators and human beings
> are notoriously bad at safely keeping secrets (they either end up sharing
> them or losing/forgetting them).
>
> - Make an executive decision that CouchDB no longer has a primary use case
> for multimaster replication across administrative domains. This feature is
> always what set Couch apart for me. Replicating documents between
> decentralized administrative domains instead of only being a centralized
> document repository for a single organization. I get that folks like IBM
> and other large single organization installations really don't care about
> replicating/sharing their data with third party organizations; and that
> sharing a multimaster distributed database across administrative domains is
> not as common as a single organization with their own large private
> repository and set of users; but I really like Couch specifically for the
> cross domain replication use case. I think it's a medium term problem that
> people are looking for solutions to, and it's non-trivial to solve for.
> How can we share "records" securely 'between' many organizations instead of
> each organization trying to keep their own separate data instance copies in
> sync with each other? I think Couch, and the Couch replication protocol,
> is a leading contender in addressing that challenge.
>
> I bring this up now because I think whatever approach is used to adress the
> cross domain authorization issue will have a huge influence on the
> feature's design (alongside other factors).
>
> Thanks,
> Mike
>> On Feb 26, 2019 3:39 AM, "Jan Lehnardt" <ja...@apache.org> wrote:
>>
>> Heya Garren,
>>
>> thanks for having a look. From a code-organisation perspective, some of my
>> edits can easily live in a separate app vs. src/couch, that mostly an code
>> orga task which I’m happy to do. The epi suggestion surely helps with with
>> the handler overrides.
>>
>> Some of the changes however have to be in core CouchDB, specifically the
>> storing of _access information on the various doc records, in order to
>> ensure efficient updates. That’s not something a fully external app can
>> manage. Whether it’s an extra field on those records or rather an entry
>> in the existing meta field is secondary, but this needs propagating into
>> by-id (and maybe also by-seq).
>>
>> I’m not sure about your suggestion to listen to all access=true DBs’s
>> _changes feeds to generate the required indexes. That sounds like building
>> a new mini couch_mrview/couch_index rather than re-using that
>> infrastructure
>> with minimal edits.
>>
>> As for the FDB option, going through the code this far helped me understand
>> all the building blocks required and I think adding this to FDB CouchDB
>> would maybe take a week total (i.e. be significantly easier), so I’m not
>> aiming to re-use much for that implementation other than the future test
>> suite.
>>
>> That said, I’m very not married to my existing code, and I’d love to hear
>> any and all ways to simplify things.
>>
>> Best
>> Jan
>> —
>>
>>
>>> On 26. Feb 2019, at 11:18, Garren Smith <ga...@apache.org> wrote:
>>>
>>> Hi Jan,
>>>
>>> I've been giving this some thought and I wonder if we should take a step
>>> back and rethink how we do this. Instead of implementing this directly
>> into
>>> the CouchDB core code, it might be better to write this as an application
>>> similar to Dreyfus - Cloudant's search[1]. Instead of writing this code
>>> directly in the core CouchDB code rather we write this as another
>>> application. I'm hoping then that you wouldn't have to make huge
>>> modifications to the CouchDB codebase which should make this easier to
>> do.
>>> The application would override the _all_docs and _changes endpoints, and
>> if
>>> a user has enabled access=true for that database then you could then
>> return
>>> the _all_docs and _changes requests from your application. The epi http
>>> work is pretty fancy I think we could do some cool things around that to
>>> make this work well. The app would listen to the changes feeds of any
>>> database that has access=true and then implement the required index's for
>>> _all_docs and changes. I think we then would not have to create a custom
>>> indexer as we could build the indexes when new changes arrive.
>>>
>>> I'm also hoping that another advantage of doing this as an app that
>> listens
>>> to the changes feed is that there should be minimal work to get this to
>>> work when we switch to fdb.
>>>
>>> This is obviously just an idea I had and I thought I would share it, not
>> in
>>> an attempt to derail what you doing, but hopefully in an attempt to make
>>> sure we find the easiest and most effective way to get this done.
>>>
>>> Cheers
>>> Garren
>>>
>>>
>>> [1] https://github.com/cloudant-labs/dreyfus
>>>
>>>>> On Sun, Feb 17, 2019 at 4:25 PM Jan Lehnardt <ja...@apache.org> wrote:
>>>>
>>>> Hi Everyone,
>>>>
>>>> I’m happy to share my work in progress attempt to implement the per-doc
>>>> access control feature we discussed a good while ago:
>>>>
>>>>
>>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb
>> 509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>>>> <
>>>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb
>> 509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>>>>
>>>> You can check out my branch here:
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1 <
>>>> https://github.com/apache/couchdb/compare/access?expand=1>
>>>>
>>>> It is very much work in progress, but it is far enough along to warrant
>>>> discussion.
>>>>
>>>> The main point of this branch is to show all the places that we would
>> need
>>>> to change to support the proposal.
>>>>
>>>> Things I’ve left for later:
>>>>
>>>> - currently only the first element in the _access array is used. Our
>>>> and/or syntax can be added later.
>>>> - building per-access views has not been implemented yet, couch_index
>>>> would have to be taught about the new per-access-id index.
>>>> - pretty HTTP error handling
>>>> - tests except for a tiny shell script 😇
>>>>
>>>> Implementation notes:
>>>>
>>>> You create a database with the _access feature turned on like so: PUT
>>>> /db?access=true
>>>>
>>>> I started out with storing _access in the document body, as that would
>>>> allow for a minimal change set, however, on doc updates, we try hard
>> not to
>>>> load the old doc body from the database, and forcing us to do so for
>> EVERY
>>>> doc update under _access seemed prohibitive, so I extended the #doc,
>>>> #doc_info and #full_doc_info records with a new `access` attribute that
>> is
>>>> stored in both by-id and by-seq. I will need guidance on how extending
>>>> these records impact multi-version cluster interop. And especially
>> whether
>>>> this is an acceptable approach.
>>>>
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-
>> 904ab7473ff8ddd07ea44aca414e3a36
>>>>
>>>> * * *
>>>>
>>>> The main addition is a new native query server called
>>>> couch_access_native_proc, which implements two new indexes by-access-id
>> and
>>>> by-access-seq which do what you’d expect, pass in a userCtx and retrieve
>>>> the equivalent of _all_docs or _changes, but only including those docs
>> that
>>>> match the username and roles in their _access property. The existing
>>>> handlers for _all_docs and _changes have been augmented to use the new
>>>> indexes instead of the default ones, unless the user is an admin.
>>>>
>>>>
>>>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-
>> fbb53323f07579be5e46ba63cb6701c4
>>>>
>>>> * * *
>>>>
>>>> The rest of the diff is concerned with making document CRUD behave as
>>>> you’d expect it. See this little demonstration for what things look
>> like:
>>>>
>>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <
>>>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m
>> just
>>>> noticing that there might be something wonky with DELETE, but you’ll get
>>>> the gist #rimshot)
>>>>
>>>> * * *
>>>>
>>>> Open questions:
>>>>
>>>> - The aim of this is to get as close to regular CouchDB behaviour as
>>>> possible. One thing that is new however which would require all apps to
>> be
>>>> changed is that for an _access enabled database to include an _access
>> field
>>>> in their docs (docs with no _access are admin-only for now). We might
>> want
>>>> to consider on new document writes to auto-insert the authenticated
>> user’s
>>>> name as the first element in the _access array, so existing apps “just
>>>> work”.
>>>>
>>>> - Interplay with partitioned dbs: eschewing db-per-user is already a
>> large
>>>> boon if you have a lot of users, but making those per-user requests
>> inside
>>>> an _access enabled database efficient would be doubly nice, so why not
>> use
>>>> the username from the first question above and use that as the partition
>>>> key? This would work nicely for natural users with their own docs that
>> want
>>>> to share them with others later, but I can easily imagine a pipelined
>> use
>>>> of CouchDB, where a “collector” user creates all new docs, an “analyser”
>>>> takes them over and hand them to a “result” user for viewing. In that
>> case,
>>>> we’d violate the high-cardinality rule of partitions (have a lot of
>> small
>>>> ones), instead all docs go through all three users. I’d be okay with
>>>> treating the later scenario as a minor use-case, but for that use-case,
>> we
>>>> should be able to disable auto-partitioning on db creation.
>>>>
>>>> - building access view indexes for docs that have frequent _access
>>>> changes, lead to many orphaned view indexes, we should look at an
>>>> auto-cleanup solution here (maybe keep 1-N indexes in case folks just
>> swap
>>>> back and forth).
>>>>
>>>> * * *
>>>>
>>>> I’ll leave this here for now, I’m sure there are a few more things to
>>>> consider.
>>>>
>>>> I’d love to hear any and all feedback you might have. Especially if
>>>> anything is unclear.
>>>>
>>>> Best
>>>> Jan
>>>> —
>>
>> --
>> Professional Support for Apache CouchDB:
>> https://neighbourhood.ie/couchdb-support/
>>
>>
Re: [DISCUSS] Per-doc access control
Posted by Michael Fair <mi...@daclubhouse.net>.
One thing I've always been concerned about when it comes to "user based"
document access is how it interacts with cross domain replication, and
mobile replication (or "user sliced" replication).
I believe this requires some kind of concept to a "Decentralized ID", or
mapping user ids (roles) per database to system users (users) because the
idea of a "user" and who the set of users are across administrative domains
isn't likely to be the same set.
Picture the same database replicating with each other between two
adminstrative domains; like say your domain and my domain ran two separate
Couch instances and we shared a common database. For example, a "product
catalog", with per user access controls turned on so your organization can
edit products and their descriptions and prices and mine could create
purchase orders, and we both can update "tickets/issues/returns".
I think its super important to figure out how this case ought to behave
when considering the user access design. We will most likely have
different user sets in our Couch server installations.
I suggest considering that all document databases only have the concept of
role ids, and not user ids, internal to the database and that privileges be
granted to those roles. It's a requirement to create both a database role
and a system user and link the two to get access to the database; the
default role is called "public" which by default has read/write access to
all documents and users are automatically linked/mapped to it.
It's also fine to name a database role the same as a system userid (e.g.
the 'mfair' role); but that the database not have a concept of a "user"
(authentication) only the "role" (authorization). The system daemon
handles autheticating users and there's a mapping of system users to
database roles (like the 'mfair' system user could map to the 'mfair'
database role). The roles (and access privileges) would replicate with the
database but the system users would not.
This isn't a completely baked thought yet, but I think it goes in the right
direction.
Each independent server in a separate administrative domain would have its
own set of users, each server then has to map those system users to
database roles. This should only have to be done once.
Couch servers could also choose to replicate their user databases too.
Multiple Couch servers that replicate their user databases with each other
is what I'm calling an "administrative domain" or simply "domain" for short.
A user can also be mapped to (i.e. hold) multiple roles in the same
database simultaneously.
This model is very close to how MS SQL Server and other SQL databases
handles users.
..........
Other ideas include:
- Disclaim that replication with other domains simply breaks user access
controls. It's too complicated so its not supported.
- Replication can only be done within the context of a role that has
read/write access to the database's documents (a new replication
parameter). Authenticating this login for automated replication might be
tricky (remote user credentials stored in the replication database?)...
- putting all the users inside the database itself so that those ids
replicate in addition to the contents but then every server has to be
entrusted to authenticate every user in every database it shares with other
domains amd the set of users becomes the superset of all users across all
participating domains. Bring able to reset a user's password in another
domain because you have access to manipulating the database's users seems
"wrong"...
- Using a decentralized p2p identity scheme like pki and using Couch itself
as a distributed public key store. This has the advantage that docs can be
encrypted and decryption secrets protected by keypairs so remote databases
can't automatically read contents they shouldn't... It's obviously more
complicated than simply trusting the remote administrators and human beings
are notoriously bad at safely keeping secrets (they either end up sharing
them or losing/forgetting them).
- Make an executive decision that CouchDB no longer has a primary use case
for multimaster replication across administrative domains. This feature is
always what set Couch apart for me. Replicating documents between
decentralized administrative domains instead of only being a centralized
document repository for a single organization. I get that folks like IBM
and other large single organization installations really don't care about
replicating/sharing their data with third party organizations; and that
sharing a multimaster distributed database across administrative domains is
not as common as a single organization with their own large private
repository and set of users; but I really like Couch specifically for the
cross domain replication use case. I think it's a medium term problem that
people are looking for solutions to, and it's non-trivial to solve for.
How can we share "records" securely 'between' many organizations instead of
each organization trying to keep their own separate data instance copies in
sync with each other? I think Couch, and the Couch replication protocol,
is a leading contender in addressing that challenge.
I bring this up now because I think whatever approach is used to adress the
cross domain authorization issue will have a huge influence on the
feature's design (alongside other factors).
Thanks,
Mike
On Feb 26, 2019 3:39 AM, "Jan Lehnardt" <ja...@apache.org> wrote:
> Heya Garren,
>
> thanks for having a look. From a code-organisation perspective, some of my
> edits can easily live in a separate app vs. src/couch, that mostly an code
> orga task which I’m happy to do. The epi suggestion surely helps with with
> the handler overrides.
>
> Some of the changes however have to be in core CouchDB, specifically the
> storing of _access information on the various doc records, in order to
> ensure efficient updates. That’s not something a fully external app can
> manage. Whether it’s an extra field on those records or rather an entry
> in the existing meta field is secondary, but this needs propagating into
> by-id (and maybe also by-seq).
>
> I’m not sure about your suggestion to listen to all access=true DBs’s
> _changes feeds to generate the required indexes. That sounds like building
> a new mini couch_mrview/couch_index rather than re-using that
> infrastructure
> with minimal edits.
>
> As for the FDB option, going through the code this far helped me understand
> all the building blocks required and I think adding this to FDB CouchDB
> would maybe take a week total (i.e. be significantly easier), so I’m not
> aiming to re-use much for that implementation other than the future test
> suite.
>
> That said, I’m very not married to my existing code, and I’d love to hear
> any and all ways to simplify things.
>
> Best
> Jan
> —
>
>
> > On 26. Feb 2019, at 11:18, Garren Smith <ga...@apache.org> wrote:
> >
> > Hi Jan,
> >
> > I've been giving this some thought and I wonder if we should take a step
> > back and rethink how we do this. Instead of implementing this directly
> into
> > the CouchDB core code, it might be better to write this as an application
> > similar to Dreyfus - Cloudant's search[1]. Instead of writing this code
> > directly in the core CouchDB code rather we write this as another
> > application. I'm hoping then that you wouldn't have to make huge
> > modifications to the CouchDB codebase which should make this easier to
> do.
> > The application would override the _all_docs and _changes endpoints, and
> if
> > a user has enabled access=true for that database then you could then
> return
> > the _all_docs and _changes requests from your application. The epi http
> > work is pretty fancy I think we could do some cool things around that to
> > make this work well. The app would listen to the changes feeds of any
> > database that has access=true and then implement the required index's for
> > _all_docs and changes. I think we then would not have to create a custom
> > indexer as we could build the indexes when new changes arrive.
> >
> > I'm also hoping that another advantage of doing this as an app that
> listens
> > to the changes feed is that there should be minimal work to get this to
> > work when we switch to fdb.
> >
> > This is obviously just an idea I had and I thought I would share it, not
> in
> > an attempt to derail what you doing, but hopefully in an attempt to make
> > sure we find the easiest and most effective way to get this done.
> >
> > Cheers
> > Garren
> >
> >
> > [1] https://github.com/cloudant-labs/dreyfus
> >
> > On Sun, Feb 17, 2019 at 4:25 PM Jan Lehnardt <ja...@apache.org> wrote:
> >
> >> Hi Everyone,
> >>
> >> I’m happy to share my work in progress attempt to implement the per-doc
> >> access control feature we discussed a good while ago:
> >>
> >>
> >> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb
> 509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> >> <
> >> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb
> 509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> >>>
> >>
> >> You can check out my branch here:
> >>
> >> https://github.com/apache/couchdb/compare/access?expand=1 <
> >> https://github.com/apache/couchdb/compare/access?expand=1>
> >>
> >> It is very much work in progress, but it is far enough along to warrant
> >> discussion.
> >>
> >> The main point of this branch is to show all the places that we would
> need
> >> to change to support the proposal.
> >>
> >> Things I’ve left for later:
> >>
> >> - currently only the first element in the _access array is used. Our
> >> and/or syntax can be added later.
> >> - building per-access views has not been implemented yet, couch_index
> >> would have to be taught about the new per-access-id index.
> >> - pretty HTTP error handling
> >> - tests except for a tiny shell script 😇
> >>
> >> Implementation notes:
> >>
> >> You create a database with the _access feature turned on like so: PUT
> >> /db?access=true
> >>
> >> I started out with storing _access in the document body, as that would
> >> allow for a minimal change set, however, on doc updates, we try hard
> not to
> >> load the old doc body from the database, and forcing us to do so for
> EVERY
> >> doc update under _access seemed prohibitive, so I extended the #doc,
> >> #doc_info and #full_doc_info records with a new `access` attribute that
> is
> >> stored in both by-id and by-seq. I will need guidance on how extending
> >> these records impact multi-version cluster interop. And especially
> whether
> >> this is an acceptable approach.
> >>
> >>
> >> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-
> 904ab7473ff8ddd07ea44aca414e3a36
> >>
> >> * * *
> >>
> >> The main addition is a new native query server called
> >> couch_access_native_proc, which implements two new indexes by-access-id
> and
> >> by-access-seq which do what you’d expect, pass in a userCtx and retrieve
> >> the equivalent of _all_docs or _changes, but only including those docs
> that
> >> match the username and roles in their _access property. The existing
> >> handlers for _all_docs and _changes have been augmented to use the new
> >> indexes instead of the default ones, unless the user is an admin.
> >>
> >>
> >> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-
> fbb53323f07579be5e46ba63cb6701c4
> >>
> >> * * *
> >>
> >> The rest of the diff is concerned with making document CRUD behave as
> >> you’d expect it. See this little demonstration for what things look
> like:
> >>
> >> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <
> >> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m
> just
> >> noticing that there might be something wonky with DELETE, but you’ll get
> >> the gist #rimshot)
> >>
> >> * * *
> >>
> >> Open questions:
> >>
> >> - The aim of this is to get as close to regular CouchDB behaviour as
> >> possible. One thing that is new however which would require all apps to
> be
> >> changed is that for an _access enabled database to include an _access
> field
> >> in their docs (docs with no _access are admin-only for now). We might
> want
> >> to consider on new document writes to auto-insert the authenticated
> user’s
> >> name as the first element in the _access array, so existing apps “just
> >> work”.
> >>
> >> - Interplay with partitioned dbs: eschewing db-per-user is already a
> large
> >> boon if you have a lot of users, but making those per-user requests
> inside
> >> an _access enabled database efficient would be doubly nice, so why not
> use
> >> the username from the first question above and use that as the partition
> >> key? This would work nicely for natural users with their own docs that
> want
> >> to share them with others later, but I can easily imagine a pipelined
> use
> >> of CouchDB, where a “collector” user creates all new docs, an “analyser”
> >> takes them over and hand them to a “result” user for viewing. In that
> case,
> >> we’d violate the high-cardinality rule of partitions (have a lot of
> small
> >> ones), instead all docs go through all three users. I’d be okay with
> >> treating the later scenario as a minor use-case, but for that use-case,
> we
> >> should be able to disable auto-partitioning on db creation.
> >>
> >> - building access view indexes for docs that have frequent _access
> >> changes, lead to many orphaned view indexes, we should look at an
> >> auto-cleanup solution here (maybe keep 1-N indexes in case folks just
> swap
> >> back and forth).
> >>
> >> * * *
> >>
> >> I’ll leave this here for now, I’m sure there are a few more things to
> >> consider.
> >>
> >> I’d love to hear any and all feedback you might have. Especially if
> >> anything is unclear.
> >>
> >> Best
> >> Jan
> >> —
>
> --
> Professional Support for Apache CouchDB:
> https://neighbourhood.ie/couchdb-support/
>
>
Re: [DISCUSS] Per-doc access control
Posted by Jan Lehnardt <ja...@apache.org>.
Heya Garren,
thanks for having a look. From a code-organisation perspective, some of my
edits can easily live in a separate app vs. src/couch, that mostly an code
orga task which I’m happy to do. The epi suggestion surely helps with with
the handler overrides.
Some of the changes however have to be in core CouchDB, specifically the
storing of _access information on the various doc records, in order to
ensure efficient updates. That’s not something a fully external app can
manage. Whether it’s an extra field on those records or rather an entry
in the existing meta field is secondary, but this needs propagating into
by-id (and maybe also by-seq).
I’m not sure about your suggestion to listen to all access=true DBs’s
_changes feeds to generate the required indexes. That sounds like building
a new mini couch_mrview/couch_index rather than re-using that infrastructure
with minimal edits.
As for the FDB option, going through the code this far helped me understand
all the building blocks required and I think adding this to FDB CouchDB
would maybe take a week total (i.e. be significantly easier), so I’m not
aiming to re-use much for that implementation other than the future test
suite.
That said, I’m very not married to my existing code, and I’d love to hear
any and all ways to simplify things.
Best
Jan
—
> On 26. Feb 2019, at 11:18, Garren Smith <ga...@apache.org> wrote:
>
> Hi Jan,
>
> I've been giving this some thought and I wonder if we should take a step
> back and rethink how we do this. Instead of implementing this directly into
> the CouchDB core code, it might be better to write this as an application
> similar to Dreyfus - Cloudant's search[1]. Instead of writing this code
> directly in the core CouchDB code rather we write this as another
> application. I'm hoping then that you wouldn't have to make huge
> modifications to the CouchDB codebase which should make this easier to do.
> The application would override the _all_docs and _changes endpoints, and if
> a user has enabled access=true for that database then you could then return
> the _all_docs and _changes requests from your application. The epi http
> work is pretty fancy I think we could do some cool things around that to
> make this work well. The app would listen to the changes feeds of any
> database that has access=true and then implement the required index's for
> _all_docs and changes. I think we then would not have to create a custom
> indexer as we could build the indexes when new changes arrive.
>
> I'm also hoping that another advantage of doing this as an app that listens
> to the changes feed is that there should be minimal work to get this to
> work when we switch to fdb.
>
> This is obviously just an idea I had and I thought I would share it, not in
> an attempt to derail what you doing, but hopefully in an attempt to make
> sure we find the easiest and most effective way to get this done.
>
> Cheers
> Garren
>
>
> [1] https://github.com/cloudant-labs/dreyfus
>
> On Sun, Feb 17, 2019 at 4:25 PM Jan Lehnardt <ja...@apache.org> wrote:
>
>> Hi Everyone,
>>
>> I’m happy to share my work in progress attempt to implement the per-doc
>> access control feature we discussed a good while ago:
>>
>>
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>> <
>> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
>>>
>>
>> You can check out my branch here:
>>
>> https://github.com/apache/couchdb/compare/access?expand=1 <
>> https://github.com/apache/couchdb/compare/access?expand=1>
>>
>> It is very much work in progress, but it is far enough along to warrant
>> discussion.
>>
>> The main point of this branch is to show all the places that we would need
>> to change to support the proposal.
>>
>> Things I’ve left for later:
>>
>> - currently only the first element in the _access array is used. Our
>> and/or syntax can be added later.
>> - building per-access views has not been implemented yet, couch_index
>> would have to be taught about the new per-access-id index.
>> - pretty HTTP error handling
>> - tests except for a tiny shell script 😇
>>
>> Implementation notes:
>>
>> You create a database with the _access feature turned on like so: PUT
>> /db?access=true
>>
>> I started out with storing _access in the document body, as that would
>> allow for a minimal change set, however, on doc updates, we try hard not to
>> load the old doc body from the database, and forcing us to do so for EVERY
>> doc update under _access seemed prohibitive, so I extended the #doc,
>> #doc_info and #full_doc_info records with a new `access` attribute that is
>> stored in both by-id and by-seq. I will need guidance on how extending
>> these records impact multi-version cluster interop. And especially whether
>> this is an acceptable approach.
>>
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>>
>> * * *
>>
>> The main addition is a new native query server called
>> couch_access_native_proc, which implements two new indexes by-access-id and
>> by-access-seq which do what you’d expect, pass in a userCtx and retrieve
>> the equivalent of _all_docs or _changes, but only including those docs that
>> match the username and roles in their _access property. The existing
>> handlers for _all_docs and _changes have been augmented to use the new
>> indexes instead of the default ones, unless the user is an admin.
>>
>>
>> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>>
>> * * *
>>
>> The rest of the diff is concerned with making document CRUD behave as
>> you’d expect it. See this little demonstration for what things look like:
>>
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <
>> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just
>> noticing that there might be something wonky with DELETE, but you’ll get
>> the gist #rimshot)
>>
>> * * *
>>
>> Open questions:
>>
>> - The aim of this is to get as close to regular CouchDB behaviour as
>> possible. One thing that is new however which would require all apps to be
>> changed is that for an _access enabled database to include an _access field
>> in their docs (docs with no _access are admin-only for now). We might want
>> to consider on new document writes to auto-insert the authenticated user’s
>> name as the first element in the _access array, so existing apps “just
>> work”.
>>
>> - Interplay with partitioned dbs: eschewing db-per-user is already a large
>> boon if you have a lot of users, but making those per-user requests inside
>> an _access enabled database efficient would be doubly nice, so why not use
>> the username from the first question above and use that as the partition
>> key? This would work nicely for natural users with their own docs that want
>> to share them with others later, but I can easily imagine a pipelined use
>> of CouchDB, where a “collector” user creates all new docs, an “analyser”
>> takes them over and hand them to a “result” user for viewing. In that case,
>> we’d violate the high-cardinality rule of partitions (have a lot of small
>> ones), instead all docs go through all three users. I’d be okay with
>> treating the later scenario as a minor use-case, but for that use-case, we
>> should be able to disable auto-partitioning on db creation.
>>
>> - building access view indexes for docs that have frequent _access
>> changes, lead to many orphaned view indexes, we should look at an
>> auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap
>> back and forth).
>>
>> * * *
>>
>> I’ll leave this here for now, I’m sure there are a few more things to
>> consider.
>>
>> I’d love to hear any and all feedback you might have. Especially if
>> anything is unclear.
>>
>> Best
>> Jan
>> —
--
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/
Re: [DISCUSS] Per-doc access control
Posted by Garren Smith <ga...@apache.org>.
Hi Jan,
I've been giving this some thought and I wonder if we should take a step
back and rethink how we do this. Instead of implementing this directly into
the CouchDB core code, it might be better to write this as an application
similar to Dreyfus - Cloudant's search[1]. Instead of writing this code
directly in the core CouchDB code rather we write this as another
application. I'm hoping then that you wouldn't have to make huge
modifications to the CouchDB codebase which should make this easier to do.
The application would override the _all_docs and _changes endpoints, and if
a user has enabled access=true for that database then you could then return
the _all_docs and _changes requests from your application. The epi http
work is pretty fancy I think we could do some cool things around that to
make this work well. The app would listen to the changes feeds of any
database that has access=true and then implement the required index's for
_all_docs and changes. I think we then would not have to create a custom
indexer as we could build the indexes when new changes arrive.
I'm also hoping that another advantage of doing this as an app that listens
to the changes feed is that there should be minimal work to get this to
work when we switch to fdb.
This is obviously just an idea I had and I thought I would share it, not in
an attempt to derail what you doing, but hopefully in an attempt to make
sure we find the easiest and most effective way to get this done.
Cheers
Garren
[1] https://github.com/cloudant-labs/dreyfus
On Sun, Feb 17, 2019 at 4:25 PM Jan Lehnardt <ja...@apache.org> wrote:
> Hi Everyone,
>
> I’m happy to share my work in progress attempt to implement the per-doc
> access control feature we discussed a good while ago:
>
>
> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> <
> https://lists.apache.org/thread.html/6aa77dd8e5974a3a540758c6902ccb509ab5a2e4802ecf4fd724a5e4@%3Cdev.couchdb.apache.org%3E
> >
>
> You can check out my branch here:
>
> https://github.com/apache/couchdb/compare/access?expand=1 <
> https://github.com/apache/couchdb/compare/access?expand=1>
>
> It is very much work in progress, but it is far enough along to warrant
> discussion.
>
> The main point of this branch is to show all the places that we would need
> to change to support the proposal.
>
> Things I’ve left for later:
>
> - currently only the first element in the _access array is used. Our
> and/or syntax can be added later.
> - building per-access views has not been implemented yet, couch_index
> would have to be taught about the new per-access-id index.
> - pretty HTTP error handling
> - tests except for a tiny shell script 😇
>
> Implementation notes:
>
> You create a database with the _access feature turned on like so: PUT
> /db?access=true
>
> I started out with storing _access in the document body, as that would
> allow for a minimal change set, however, on doc updates, we try hard not to
> load the old doc body from the database, and forcing us to do so for EVERY
> doc update under _access seemed prohibitive, so I extended the #doc,
> #doc_info and #full_doc_info records with a new `access` attribute that is
> stored in both by-id and by-seq. I will need guidance on how extending
> these records impact multi-version cluster interop. And especially whether
> this is an acceptable approach.
>
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-904ab7473ff8ddd07ea44aca414e3a36
>
> * * *
>
> The main addition is a new native query server called
> couch_access_native_proc, which implements two new indexes by-access-id and
> by-access-seq which do what you’d expect, pass in a userCtx and retrieve
> the equivalent of _all_docs or _changes, but only including those docs that
> match the username and roles in their _access property. The existing
> handlers for _all_docs and _changes have been augmented to use the new
> indexes instead of the default ones, unless the user is an admin.
>
>
> https://github.com/apache/couchdb/compare/access?expand=1&ws=0#diff-fbb53323f07579be5e46ba63cb6701c4
>
> * * *
>
> The rest of the diff is concerned with making document CRUD behave as
> you’d expect it. See this little demonstration for what things look like:
>
> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497 <
> https://gist.github.com/janl/b6d3f7502aa20b7b9ab9d9dcb8e92497> (I’m just
> noticing that there might be something wonky with DELETE, but you’ll get
> the gist #rimshot)
>
> * * *
>
> Open questions:
>
> - The aim of this is to get as close to regular CouchDB behaviour as
> possible. One thing that is new however which would require all apps to be
> changed is that for an _access enabled database to include an _access field
> in their docs (docs with no _access are admin-only for now). We might want
> to consider on new document writes to auto-insert the authenticated user’s
> name as the first element in the _access array, so existing apps “just
> work”.
>
> - Interplay with partitioned dbs: eschewing db-per-user is already a large
> boon if you have a lot of users, but making those per-user requests inside
> an _access enabled database efficient would be doubly nice, so why not use
> the username from the first question above and use that as the partition
> key? This would work nicely for natural users with their own docs that want
> to share them with others later, but I can easily imagine a pipelined use
> of CouchDB, where a “collector” user creates all new docs, an “analyser”
> takes them over and hand them to a “result” user for viewing. In that case,
> we’d violate the high-cardinality rule of partitions (have a lot of small
> ones), instead all docs go through all three users. I’d be okay with
> treating the later scenario as a minor use-case, but for that use-case, we
> should be able to disable auto-partitioning on db creation.
>
> - building access view indexes for docs that have frequent _access
> changes, lead to many orphaned view indexes, we should look at an
> auto-cleanup solution here (maybe keep 1-N indexes in case folks just swap
> back and forth).
>
> * * *
>
> I’ll leave this here for now, I’m sure there are a few more things to
> consider.
>
> I’d love to hear any and all feedback you might have. Especially if
> anything is unclear.
>
> Best
> Jan
> —