You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Marcel Reutegger <mr...@adobe.com> on 2014/02/19 17:45:59 UTC
permission store & rep:modCount
hi,
I ran into a case where multiple sessions on different cluster nodes
create access control entries for the same principal. admittedly, this
was done in a test and may not be too likely reality, but it showed
a problem with the permission store and how clustering on mongo
works.
it seems the node for the principal in the permission store is updated
whenever there is a change. that is, I see frequent changes on the
rep:modCount on /jcr:system/rep:permissionStore/crx.default/everyone
even when there are no other changes on that node. I might be misreading
the data, but I was wondering what the purpose of this rep:modCount
is?
with the DocumentNodeStore on MongoDB in a cluster, the rep:ModCount
can become a problem when there are concurrent updates for a principal.
within one DocumentNodeStore instance there is a fallback similar to the
pessimistic locking in the segment implementation, but clustered DocumentNodeStore
instances do not have a lock mechanism to serialize writes across the complete cluster.
this is considered too expensive. in the clustered case, the commit is simply
retried a number of times but at some point the save call fails if retries are still
not successful.
regards
marcel
RE: permission store & rep:modCount
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
> another approach would be to store the entire permission data of a
> principal in 1 large property (eg json serialized). but then you'll
> see update of that property whenever you change the permissions of a
> principal. would that perform better?
no, that would be even worse, because then you are definitively forced to
serialize updates. with the current structure it is at least possible to perform
concurrent updates (apart from the rep:modCount).
Regards
Marcel
RE: permission store & rep:modCount
Posted by Marcel Reutegger <mr...@adobe.com>.
> > no, you would only use a small number like 8. this is similar to how other
> > concurrent data structures work. e.g. the ConcurrentHashMap does not
> > have a single field for the size. it keeps the size per segment and therefore
> > allows updates to happen concurrently on different segments.
>
> Don't these data-structures use locking for each segment to ensure
> consistency? How would you ensure that the same problem doesn't occur
> on the per-segment mod count then? AFAIU, without locking we only
> reduce the probability of the problem happening. Is that a
> satisfactory solution?
ideally there wouldn't be any modCount. but you are right, a per-segment mod
count doesn't solve the problem entirely, it only reduces the probability.
it would be possible to 'guarantee' an update of the permission store with
infinite retries, but then we may run into a liveness problem. what if one update
always loses because of a conflict?
Regards
Marcel
Re: permission store & rep:modCount
Posted by Julian Sedding <js...@gmail.com>.
Hi Marcel
> no, you would only use a small number like 8. this is similar to how other
> concurrent data structures work. e.g. the ConcurrentHashMap does not
> have a single field for the size. it keeps the size per segment and therefore
> allows updates to happen concurrently on different segments.
Don't these data-structures use locking for each segment to ensure
consistency? How would you ensure that the same problem doesn't occur
on the per-segment mod count then? AFAIU, without locking we only
reduce the probability of the problem happening. Is that a
satisfactory solution?
Regards
Julian
On Tue, Feb 25, 2014 at 8:42 AM, Marcel Reutegger <mr...@adobe.com> wrote:
> Hi,
>
>> maybe I don't understand - but you need to sync anyways when you write
>> ACLs. or when do you detect collisions? the permission store itself is
>> a problem for concurrency, as sparse modifications are consolidated in
>> 1 place. so even if 1 node updates ACLs on /content/foo and the other
>> on /content/bar, both need to update /<permissionstore>/everyone.
>
> but in most cases these updates could happen concurrently, at least with
> the DocumentNodeStore (aka MongoMK). but the update on the mod
> count causes a conflict. AFAIU the rest of the update is usually non-conflicting.
> the permission hook distributes the access control information into buckets
> under /<permissionstore>/everyone.
>
>> > so, what if we replace the single mod-count property with multiple
>> > properties? based on the hash of the modified access control entry
>> > you would update one of them. this would at least allow some concurrent
>> > updates.
>> so, if I have 1mio ACLs if have 1mio modcounts?
>
> no, you would only use a small number like 8. this is similar to how other
> concurrent data structures work. e.g. the ConcurrentHashMap does not
> have a single field for the size. it keeps the size per segment and therefore
> allows updates to happen concurrently on different segments.
>
>> as I said, if we could expose a revision number or content hash of
>> subtrees, the modCount is not needed.
>> or we disable the memory ACL cache completely on clustered environments.
>
> what is the performance impact if we do that?
>
> regards
> marcel
RE: permission store & rep:modCount
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
> maybe I don't understand - but you need to sync anyways when you write
> ACLs. or when do you detect collisions? the permission store itself is
> a problem for concurrency, as sparse modifications are consolidated in
> 1 place. so even if 1 node updates ACLs on /content/foo and the other
> on /content/bar, both need to update /<permissionstore>/everyone.
but in most cases these updates could happen concurrently, at least with
the DocumentNodeStore (aka MongoMK). but the update on the mod
count causes a conflict. AFAIU the rest of the update is usually non-conflicting.
the permission hook distributes the access control information into buckets
under /<permissionstore>/everyone.
> > so, what if we replace the single mod-count property with multiple
> > properties? based on the hash of the modified access control entry
> > you would update one of them. this would at least allow some concurrent
> > updates.
> so, if I have 1mio ACLs if have 1mio modcounts?
no, you would only use a small number like 8. this is similar to how other
concurrent data structures work. e.g. the ConcurrentHashMap does not
have a single field for the size. it keeps the size per segment and therefore
allows updates to happen concurrently on different segments.
> as I said, if we could expose a revision number or content hash of
> subtrees, the modCount is not needed.
> or we disable the memory ACL cache completely on clustered environments.
what is the performance impact if we do that?
regards
marcel
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@adobe.com>.
Hi,
I created a test that created a node structure with 10k node, each
having 1 ACL and 1 ACE for the everyone principle.
then, with concurrent reader sessions, I read the nodes randomly.
Benchmarks: ConcurrentEveryoneACLTest (oak 0.19-SNAPSHOT with everyone cache)
Fixtures: Oak-Tar
Runtime: 5
Num Items: 1000
Concurrency: 1,2,4,8,10,15,20,50
--------------------------------------
Executing benchmarks as admin: false on Oak-Tar
-----------------------------------------------------------
# ConcurrentEveryoneACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Tar , 1, 11, 12, 13, 14,
19, 391
Oak-Tar , 2, 12, 13, 14, 17,
27, 675
Oak-Tar , 4, 15, 16, 19, 23,
55, 1020
Oak-Tar , 8, 31, 50, 53, 56,
114, 743
Oak-Tar , 10, 56, 66, 70, 76,
142, 708
Oak-Tar , 15, 74, 104, 110, 117,
132, 687
Oak-Tar , 20, 92, 136, 145, 154,
169, 698
Oak-Tar , 50, 156, 353, 384, 411,
476, 670
Executing benchmarks as admin: true on Oak-Tar
-----------------------------------------------------------
# ConcurrentEveryoneACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Tar , 1, 4, 5, 5, 6,
17, 900
Oak-Tar , 2, 5, 5, 6, 8,
21, 1526
Oak-Tar , 4, 6, 7, 8, 11,
26, 2318
Oak-Tar , 8, 13, 24, 26, 33,
68, 1420
Oak-Tar , 10, 19, 30, 33, 46,
79, 1395
Oak-Tar , 15, 40, 47, 52, 76,
120, 1345
Oak-Tar , 20, 36, 61, 67, 94,
145, 1395
Oak-Tar , 50, 115, 158, 179, 255,
310, 1325
then I simply disabled the gloabal everyone cache the result is really
bad (the 50 cocurrent actually never completed after 2 minutes?):
# ConcurrentEveryoneACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Tar , 1, 41, 42, 43, 49,
54, 112
Oak-Tar , 2, 56, 57, 62, 74,
125, 158
Oak-Tar , 4, 75, 96, 117, 132,
211, 171
Oak-Tar , 8, 261, 321, 346, 419,
471, 117
Oak-Tar , 10, 475, 495, 519, 6532,
6549, 70
Oak-Tar , 15, 7646, 7647, 7655, 7660,
7661, 15
Oak-Tar , 20, 26988, 26995, 27010, 27015,
27016, 20
On the bright side, the creation of the ACLs is better for higher
concurrency without the modcount:
with mod-count:
# ConcurrentWriteACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Mongo , 1, 197, 211, 227, 276,
330, 21
Oak-Mongo , 2, 210, 212, 231, 1366,
4688, 22
Oak-Mongo , 4, 200, 208, 232, 3864,
4990, 24
Oak-Mongo , 8, 201, 238, 1520, 4480,
4554, 27
Oak-Mongo , 10, 202, 216, 1584, 5286,
6671, 28
Oak-Mongo , 15, 225, 740, 3005, 7695,
8254, 28
Oak-Mongo , 20, 219, 238, 3684, 8701,
10770, 34
Oak-Mongo , 50, 3951, 7631, 12557, 16893,
17828, 52
without mod-count:
# ConcurrentWriteACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Mongo , 1, 201, 206, 219, 259,
268, 23
Oak-Mongo , 2, 185, 194, 216, 616,
4561, 24
Oak-Mongo , 4, 191, 194, 375, 2434,
3091, 20
Oak-Mongo , 8, 207, 469, 603, 2865,
3962, 38
Oak-Mongo , 10, 510, 676, 782, 1053,
1417, 65
Oak-Mongo , 15, 965, 1223, 1344, 1673,
2137, 56
Oak-Mongo , 20, 994, 1211, 1607, 2033,
2661, 66
Oak-Mongo , 50, 3359, 3496, 3947, 4655,
5559, 96
and a quick test with storing the permissions directly in the content:
# ConcurrentWriteACLTest , C, min, 10%, 50%, 90%,
max, N
Oak-Tar , 1, 30, 32, 35, 39,
48, 143
Oak-Tar , 2, 71, 77, 79, 84,
106, 126
Oak-Tar , 4, 48, 159, 161, 164,
168, 125
Oak-Tar , 8, 313, 315, 324, 340,
347, 128
Oak-Tar , 10, 384, 395, 402, 418,
419, 130
Oak-Tar , 15, 580, 601, 615, 626,
630, 135
Oak-Tar , 20, 712, 803, 821, 847,
852, 140
Oak-Tar , 50, 533, 2053, 2134, 2163,
2168, 151
I'll keep you posted...
regards, toby
Re: permission store & rep:modCount
Posted by Angela Schreiber <an...@adobe.com>.
hi
>I looked at the problem again. Removing the mod-count improves
>concurrency when writing ACLs a lot (about 2 times faster with mongo
>and 20 threads). the read-performance does not suffer much with the
>current tests - but then the tests might not be accurate. the
>permission cache is only used to the everyone permissions, so it will
>show on systems with many of those.
i created https://issues.apache.org/jira/browse/OAK-1504 such that
can cover any kind of performance drop with benchmark test.
let's have some accurate test-setup that allows us to get reliable
figures if with and without that everyone cache and the concurrency
issues associated with the mod-count.
>bottom line: for now, I'll remove the everyone cache and the mod-count
>again. after the 1.0 release, we should review the permission store
>and handling.
i would appreciate if you could write the benchmark tests upfront such
that we can actually make any kind of statement wrt effect and potentially
decide on how to deal with it.
thanks a lot
angela
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
I looked at the problem again. Removing the mod-count improves
concurrency when writing ACLs a lot (about 2 times faster with mongo
and 20 threads). the read-performance does not suffer much with the
current tests - but then the tests might not be accurate. the
permission cache is only used to the everyone permissions, so it will
show on systems with many of those.
I also looked at jukkas approach of storing the ACLs in
content/:permissions, and it improves concurrency even more (about 3
faster times), but the read performance dropped by 50%. but this was
with a stupid algorithm.
I think that we can improve the :permissions approach, but also would
need to change the way the permissions are exposed in the API/SPI.
currently they are orthogonally provided by the PermissionProvider in
so called TreePermissions. I think it could simplify a lot, if they
would be part of the Tree API with something like
Tree.getPermissions(). but this requires more refactoring.
bottom line: for now, I'll remove the everyone cache and the mod-count
again. after the 1.0 release, we should review the permission store
and handling.
regards, toby
RE: permission store & rep:modCount
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
> FWIW DocumentNodeStore maintains a property '_modCount' on a per node
> basis and it can possibly be exposed as part of NodeStore as a hidden property.
the _modCount is not sufficiently accurate. it does not take pending updates
of the _lastRev into account. we'd have to use DocumentNodeState#lastRevision.
this reflects the last modification on that node or some descendant node.
Regards
Marcel
Re: permission store & rep:modCount
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
My suggestion would be to (additionally or only) expose the current
cluster node id in some way.
That way, oak-core (mainly hooks I guess; for example index
implementations) can create properties and nodes that are known to not
conflict with changes of other cluster nodes.
Regards,
Thomas
On 04/03/14 15:37, "Chetan Mehrotra" <ch...@gmail.com> wrote:
>FWIW DocumentNodeStore maintains a property '_modCount' on a per node
>basis and it can possibly be exposed as part of NodeStore as a hidden
>property.
>
>And that can be used to maintain the cache. DocumentNodeStore itself
>internally relies on this to keep its caches in consistent state
>Chetan Mehrotra
>
>
>On Fri, Feb 28, 2014 at 7:52 AM, Jukka Zitting <ju...@gmail.com>
>wrote:
>> Hi,
>>
>> On Thu, Feb 27, 2014 at 4:46 PM, Tobias Bocanegra <tr...@apache.org>
>>wrote:
>>> the current implementation allows to read all ACLs for a given
>>> principal very efficiently at once with no false hasNode() accesses.
>>
>> Good point.
>>
>> To avoid has/getNode() misses (that can be expensive with MongoMK) in
>> the proposed alternative, it would probably be a good idea to store a
>> :hasPermissions flag property (or something similar) in each node with
>> an ACL attached to it. Such a property could even be something like
>> :permissionModCount in case in-memory caching (and cache invalidation)
>> is still needed.
>>
>> BR,
>>
>> Jukka Zitting
Re: permission store & rep:modCount
Posted by Chetan Mehrotra <ch...@gmail.com>.
FWIW DocumentNodeStore maintains a property '_modCount' on a per node
basis and it can possibly be exposed as part of NodeStore as a hidden
property.
And that can be used to maintain the cache. DocumentNodeStore itself
internally relies on this to keep its caches in consistent state
Chetan Mehrotra
On Fri, Feb 28, 2014 at 7:52 AM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Thu, Feb 27, 2014 at 4:46 PM, Tobias Bocanegra <tr...@apache.org> wrote:
>> the current implementation allows to read all ACLs for a given
>> principal very efficiently at once with no false hasNode() accesses.
>
> Good point.
>
> To avoid has/getNode() misses (that can be expensive with MongoMK) in
> the proposed alternative, it would probably be a good idea to store a
> :hasPermissions flag property (or something similar) in each node with
> an ACL attached to it. Such a property could even be something like
> :permissionModCount in case in-memory caching (and cache invalidation)
> is still needed.
>
> BR,
>
> Jukka Zitting
Re: permission store & rep:modCount
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Feb 27, 2014 at 4:46 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> the current implementation allows to read all ACLs for a given
> principal very efficiently at once with no false hasNode() accesses.
Good point.
To avoid has/getNode() misses (that can be expensive with MongoMK) in
the proposed alternative, it would probably be a good idea to store a
:hasPermissions flag property (or something similar) in each node with
an ACL attached to it. Such a property could even be something like
:permissionModCount in case in-memory caching (and cache invalidation)
is still needed.
BR,
Jukka Zitting
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
On Thu, Feb 27, 2014 at 1:02 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Tue, Feb 25, 2014 at 2:35 AM, Angela Schreiber <an...@adobe.com> wrote:
>> i tried this out last year and it was terribly slow.
>
> We were too busy at the time to dig deeper, but I'm still surprised
> about the result. Are we sure the draft implementation didn't have
> some order-of-magnitude performance bottlenecks that couldn't be
> avoided?
>
> With the current implementation, to get permissions of a principal for
> a given path, we need to read the content stored at:
>
> /jcr:system/rep:permissionStore/<principal>/<hash(/path/to/node)>/...
>
> If we instead stored the same information along the actual path being
> accessed, the permissions could be read at:
>
> /path/to/node/:permissions/<principal>/...
>
> AFAICT there should be no reason why accessing the latter would be any
> more expensive than the former, and a variety of reasons (locality of
> access, etc.) to expect it to be notably faster. If implemented like
> this, there would be no central point of contention like we now have
> with the modCount property.
the current implementation allows to read all ACLs for a given
principal very efficiently at once with no false hasNode() accesses.
as the size of the ACLs is known beforehand for each principal, it can
quickly be decided if it's worthwhile to load all ACLs for a given
principal set or not.
But you're idea is certainly worth to follow later on. so far we
focused on assemble the ACLs directly then traversing, but storing the
pre-compiled permissions in the content was IIRC never tried.
regards, toby
Re: permission store & rep:modCount
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Feb 25, 2014 at 2:35 AM, Angela Schreiber <an...@adobe.com> wrote:
> i tried this out last year and it was terribly slow.
We were too busy at the time to dig deeper, but I'm still surprised
about the result. Are we sure the draft implementation didn't have
some order-of-magnitude performance bottlenecks that couldn't be
avoided?
With the current implementation, to get permissions of a principal for
a given path, we need to read the content stored at:
/jcr:system/rep:permissionStore/<principal>/<hash(/path/to/node)>/...
If we instead stored the same information along the actual path being
accessed, the permissions could be read at:
/path/to/node/:permissions/<principal>/...
AFAICT there should be no reason why accessing the latter would be any
more expensive than the former, and a variety of reasons (locality of
access, etc.) to expect it to be notably faster. If implemented like
this, there would be no central point of contention like we now have
with the modCount property.
BR,
Jukka Zitting
Re: permission store & rep:modCount
Posted by Angela Schreiber <an...@adobe.com>.
hi jukka
i tried this out last year and it was terribly slow. that's why
we didn't do it this way.
kind regards
angela
On 24/02/14 20:46, "Jukka Zitting" <ju...@gmail.com> wrote:
>Hi,
>
>On Mon, Feb 24, 2014 at 1:27 PM, Tobias Bocanegra <tr...@apache.org>
>wrote:
>> or we once again think of a better/faster strategy to collect the ACLs
>> and abandon the permission store idea.
>
>FWIW, I still think that the approach of picking up ACLs along the way
>during path resolution should work as well or better than looking them
>up separately from the permission store.
>
>BR,
>
>Jukka Zitting
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
On Mon, Feb 24, 2014 at 11:46 AM, Jukka Zitting <ju...@gmail.com> wrote:
> On Mon, Feb 24, 2014 at 1:27 PM, Tobias Bocanegra <tr...@apache.org> wrote:
>> or we once again think of a better/faster strategy to collect the ACLs
>> and abandon the permission store idea.
>
> FWIW, I still think that the approach of picking up ACLs along the way
> during path resolution should work as well or better than looking them
> up separately from the permission store.
maybe...we improved quite a bit on the persistence layer - maybe it's
worth another try :-)
regards, toby
Re: permission store & rep:modCount
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Feb 24, 2014 at 1:27 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> or we once again think of a better/faster strategy to collect the ACLs
> and abandon the permission store idea.
FWIW, I still think that the approach of picking up ACLs along the way
during path resolution should work as well or better than looking them
up separately from the permission store.
BR,
Jukka Zitting
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi,
On Mon, Feb 24, 2014 at 2:17 AM, Marcel Reutegger <mr...@adobe.com> wrote:
>> so what the cache does, it checks if the memory representation is
>> still on par with the one in the store by comparing the mod-count. if
>> there is another way to do this, I'm more than happy to know.
>
> the main problem currently is that this requires an implementation to
> serialize writes to the permission store for a given principal.
maybe I don't understand - but you need to sync anyways when you write
ACLs. or when do you detect collisions? the permission store itself is
a problem for concurrency, as sparse modifications are consolidated in
1 place. so even if 1 node updates ACLs on /content/foo and the other
on /content/bar, both need to update /<permissionstore>/everyone.
> so, what if we replace the single mod-count property with multiple
> properties? based on the hash of the modified access control entry
> you would update one of them. this would at least allow some concurrent
> updates.
so, if I have 1mio ACLs if have 1mio modcounts?
as I said, if we could expose a revision number or content hash of
subtrees, the modCount is not needed.
or we disable the memory ACL cache completely on clustered environments.
or we once again think of a better/faster strategy to collect the ACLs
and abandon the permission store idea. OTOH, I think that ACL
modification are relatively rare compared to other updates.
regards, toby
RE: permission store & rep:modCount
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
> so what the cache does, it checks if the memory representation is
> still on par with the one in the store by comparing the mod-count. if
> there is another way to do this, I'm more than happy to know.
the main problem currently is that this requires an implementation to
serialize writes to the permission store for a given principal.
so, what if we replace the single mod-count property with multiple
properties? based on the hash of the modified access control entry
you would update one of them. this would at least allow some concurrent
updates.
Regards
Marcel
Re: permission store & rep:modCount
Posted by Tobias Bocanegra <tr...@apache.org>.
Hi Marcel,
On Wed, Feb 19, 2014 at 8:45 AM, Marcel Reutegger <mr...@adobe.com> wrote:
> I ran into a case where multiple sessions on different cluster nodes
> create access control entries for the same principal. admittedly, this
> was done in a test and may not be too likely reality, but it showed
> a problem with the permission store and how clustering on mongo
> works.
>
> it seems the node for the principal in the permission store is updated
> whenever there is a change. that is, I see frequent changes on the
> rep:modCount on /jcr:system/rep:permissionStore/crx.default/everyone
> even when there are no other changes on that node. I might be misreading
> the data, but I was wondering what the purpose of this rep:modCount
> is?
this property is used for controlling the memory cache of the
permissions of a principal. it should only change if you alter an ACL
that involves that principal. Especially for the 'everyone' principal,
it totally makes sense to cache the permissions as this principal is
involved in so many ACLs.
so what the cache does, it checks if the memory representation is
still on par with the one in the store by comparing the mod-count. if
there is another way to do this, I'm more than happy to know.
initially I thought I could use the content-hash of that subtree - but
this is IIRC specific to the MKs. and the revisions are specific to
the document stores and not localized.
> with the DocumentNodeStore on MongoDB in a cluster, the rep:ModCount
> can become a problem when there are concurrent updates for a principal.
> within one DocumentNodeStore instance there is a fallback similar to the
> pessimistic locking in the segment implementation, but clustered DocumentNodeStore
> instances do not have a lock mechanism to serialize writes across the complete cluster.
> this is considered too expensive. in the clustered case, the commit is simply
> retried a number of times but at some point the save call fails if retries are still
> not successful.
ok. so what's the best way to determine stale cache content?
another approach would be to store the entire permission data of a
principal in 1 large property (eg json serialized). but then you'll
see update of that property whenever you change the permissions of a
principal. would that perform better?
regards, toby