Scalability of JCR observation

[Michael Dürig <md...@apache.org>]

Hi,

Currently JCR observation in Oak is targeted towards scalability and 
therefore compromises on some features:

* Observation events are generated by diffing revisions. This results in 
a consolidated view where intermediated events might have cancelled out 
and the order of events is completely random.

* No support Event.getUserId() and Event.getUserData().

Since this change might create backward compatibility issues for some 
applications, Angela, Marcel, Tom and I came up with the idea to 
implement a 2nd, more backward compatible approach to observation in 
Oak. This approach would in contrast to the former one come with 
compromises in scalability. Clients could choose between the 
implementations when registering an event listener (e.g using a marker 
interface).

An implementation approach for backward compatible observation is to use 
a commit hook to record the required changes to a journal (e.g. 
/jcr:system/rep:observation). Observation listeners would then later 
generate the observation events by scrapping that journal asynchronously.

Such an approach would increase the size of the repository by a constant 
factor < 2 since changes required for observation are also written to 
the journal. Furthermore it adds a somewhat expensive commit hook for 
recording such changes. The idea is that there is no such impact as long 
as backward compatible observation is not used.

A somewhat open question is how this should work across the cluster. The 
best way is probably for cluster sync to exclude the observation journal 
and have the built by each cluster node individually.

Michael

Re: Scalability of JCR observation

[Ian Boston <ie...@tfd.co.uk>]

Hi,
Embedding ActiveMQ into a bundle that spins up just like any other bundle
is relatively simple. Configuration for most of the standard topologies can
be done with a few URIs in OSGi properties, and, in general the only thing
that needs to be exported is a service to allow consumers and producers to
get hold of a connection. See [1].

In that example, used in Sling we ran as a broker network each node
maintaining its own shared nothing DB (using ActiveMQ's own file format
under Sling home). The bundle is a bit large since it embeds ActiveMQ's
libs and dependencies, but it was relatively trouble free over about 3
years of use in both standalone and clustered deployments with no
complaints from developers, QE's or deployers. It did not go into heavy
production for other reasons.

Tests from other places have shown that AMQ is good for significant
production, but I don't know if its good for really big clusters. I suspect
that those would be heterogeneous.

I don't know enough about RabbitMQ to know if you could do the same, but I
do know implementing a distributed queue or messaging infrastructure that
really works reliably when network fabric and hardware fails is non
trivial, and a project in its own right.

If AMQ is too big for Oak, then perhaps binding to javax.jms.* and
implementing a trivial Oak JMS service would allow deployers to swap out
for their favorite JMS implementation at deployment.

Ian

1
https://github.com/ieb/nakamura/tree/master/bundles/activemq/src/main/java/org/sakaiproject/nakamura/activemq


On 15 April 2013 19:19, Bertrand Delacretaz <bd...@apache.org> wrote:

> Hi,
>
> On Mon, Apr 15, 2013 at 11:00 AM, Thomas Mueller <mu...@adobe.com>
> wrote:
> > ...I think it would be a pity if we would have to add ActiveMQ/RabbitMQ
> to
> > Oak, as it would increase implementation and configuration complexity
> > quite a lot...
>
> Note that I was suggesting that *for large clusters*.
>
> Having a simple self-contained setup a la runnable jar for
> small/demo/test/development instances is great, but I don't think it's
> a problem for a large cluster to require more tools to perform.
>
> I'm also not sure if adding the right tools increases implementation
> complexity - it's rather delegating hard stuff to projects which have
> already solved those issues.
>
> -Bertrand
>

Re: Scalability of JCR observation

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Mon, Apr 15, 2013 at 11:00 AM, Thomas Mueller <mu...@adobe.com> wrote:
> ...I think it would be a pity if we would have to add ActiveMQ/RabbitMQ to
> Oak, as it would increase implementation and configuration complexity
> quite a lot...

Note that I was suggesting that *for large clusters*.

Having a simple self-contained setup a la runnable jar for
small/demo/test/development instances is great, but I don't think it's
a problem for a large cluster to require more tools to perform.

I'm also not sure if adding the right tools increases implementation
complexity - it's rather delegating hard stuff to projects which have
already solved those issues.

-Bertrand

Re: Scalability of JCR observation

[Lukas Eder <lu...@gmail.com>]

2013/4/17 Jukka Zitting <ju...@gmail.com>:
> Hi,
>
> On Wed, Apr 17, 2013 at 3:35 AM, Ian Boston <ie...@tfd.co.uk> wrote:
>> When mentioning AMQ, I forgot to mention that one size doesn't fit all use
>> cases. When I first connected AMQ to Jackrabbit observation via OSGi
>> events, AMQ rapidly became swamped with the volume of events. OSGi events
>> were just about able to cope but as soon as the events were propagated over
>> the network the volume was too great.
>
> Indeed. Most message queue systems out there are designed for
> application/document level messages (like "document added", "action
> taken", etc.), i.e. the granularity of events is generally coarser
> than the field-level events ("property changed", "node added", etc.)
> you get with JCR observation.
>
> My thinking here is that the best approach for most use cases will be
> to use a commit hook to consolidate the low level changes to less
> granular application events that can then be processed in the
> background using JCR observation, OSGi events, a generic message queue
> system, or directly by a polling thread in the client application.

I agree with this line of thinking.

Instead of doing any performance improvement or event aggregation at a
low-level event producer site, I think that Oak itself should generate
as many meaningful events as possible. With a sensible eventing SPI,
it is easy for an Oak user to implement "event listener pipes" or
"event transformers" which filter, aggregate, and package low-level
events into fewer but bigger messages, depending on the concrete event
distribution requirements. The originally suggested aggregation on a
revision level would be an example for such a "custom" aggregation.

If MQ or OSGi event listener implementations are flooded too quickly,
Oak (or an Oak extension module) could provide a sample implementation
for event accumulation (at the producer site) and dispersion (at the
consumer site), to transparently package events into fewer but bigger
physical messages for transport - while still being able to consume
each low-level event at the consumer site if needed.

Oak itself, however, shouldn't make too many architecture / topology
assumptions about the components that build on it. I feel that
concrete implementation facts (internal and external to Oak) are
beginning to leak into Oak's eventing SPI.

Cheers,
Lukas

Re: Scalability of JCR observation

[Bertrand Delacretaz <bd...@apache.org>]

On Wed, Apr 17, 2013 at 8:36 AM, Jukka Zitting <ju...@gmail.com> wrote:
> ...My thinking here is that the best approach for most use cases will be
> to use a commit hook to consolidate the low level changes to less
> granular application events that can then be processed in the
> background using JCR observation, OSGi events, a generic message queue
> system, or directly by a polling thread in the client application....

>From my JCR user's point of view, I like that - I often find myself
merging JCR events to get a more meaningful view of actual changes and
keep load low, at the application level. Having hooks for that in Oak
sounds great, but the full event stream probably needs to optionally
be available, at least in a "JCR compliant" mode.

-Bertrand

Re: Scalability of JCR observation

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Apr 17, 2013 at 3:35 AM, Ian Boston <ie...@tfd.co.uk> wrote:
> When mentioning AMQ, I forgot to mention that one size doesn't fit all use
> cases. When I first connected AMQ to Jackrabbit observation via OSGi
> events, AMQ rapidly became swamped with the volume of events. OSGi events
> were just about able to cope but as soon as the events were propagated over
> the network the volume was too great.

Indeed. Most message queue systems out there are designed for
application/document level messages (like "document added", "action
taken", etc.), i.e. the granularity of events is generally coarser
than the field-level events ("property changed", "node added", etc.)
you get with JCR observation.

My thinking here is that the best approach for most use cases will be
to use a commit hook to consolidate the low level changes to less
granular application events that can then be processed in the
background using JCR observation, OSGi events, a generic message queue
system, or directly by a polling thread in the client application.

BR,

Jukka Zitting

Re: Scalability of JCR observation

[Ian Boston <ie...@tfd.co.uk>]

When mentioning AMQ, I forgot to mention that one size doesn't fit all use
cases. When I first connected AMQ to Jackrabbit observation via OSGi
events, AMQ rapidly became swamped with the volume of events. OSGi events
were just about able to cope but as soon as the events were propagated over
the network the volume was too great. I had to filter. Oak observa

In the unlikely event that has not been considered, it might change the
thinking about the nature of the underlying infrastructure as well
as separating high volume local messages from cluster wide distribution. I
would still plead for reusing an existing best of breed solution with the
right license and sustainability, if possible. If not, an API layer so
deployers can replace the distributed messaging part with
their preferred solution.

Ian


On 17 April 2013 00:26, Dominik Süß <do...@gmail.com> wrote:

> I see some overlap with the latest work of Carsten in Sling regarding
> Discovery API[0]. Since Sling typically should work uppon JCR / Oak it
> might be good not to follow different patterns. For a combined solution I
> do think it would be great to have one pluggable mediating system instead
> of two which might have strange sideeffects for rejoin scenarios in a
> cluster.
>
> Just my 2 cents
> Dominik
>
> [0]http://markmail.org/thread/w3kgl7jxvhki3oqj
>
>
> On Tue, Apr 16, 2013 at 11:51 AM, Michael Dürig <md...@apache.org>
> wrote:
>
> >
> >
> > On 15.4.13 9:46, Julian Reschke wrote:
> >
> >> On 2013-04-15 10:32, Bertrand Delacretaz wrote:
> >>
> >
> >  So I'm wondering if using an existing distributed message queue
> >>> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
> >>> is only a problem in very large Oak setups, so having to install
> >>> additional components might not be an issue.
> >>>
> >>
> >> Could that also help with implementing proper JCR Locking (or are we
> >> there already???).
> >>
> >
> > Probably. The idea of making external coordinaters pluggable has come up
> > before: https://issues.apache.org/**jira/browse/OAK-150?**
> > focusedCommentId=13401328&**page=com.atlassian.jira.**
> > plugin.system.issuetabpanels:**comment-tabpanel#comment-**13401328<
> https://issues.apache.org/jira/browse/OAK-150?focusedCommentId=13401328&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13401328
> >
> >
> > Michael
> >
>

Re: Scalability of JCR observation

[Stefan Egli <eg...@adobe.com>]

Hi,

On 4/16/13 4:26 PM, "Dominik Süß" <do...@gmail.com> wrote:

>I see some overlap with the latest work of Carsten in Sling regarding
>Discovery API[0]. Since Sling typically should work uppon JCR / Oak it
>might be good not to follow different patterns. For a combined solution I
>do think it would be great to have one pluggable mediating system instead
>of two which might have strange sideeffects for rejoin scenarios in a
>cluster.

+1

If there was a jms/messaging client available in oak (pluggable) that an
implementation of the discovery.api (at the sling level..) could reuse,
that would definitely result in a more reliable 'cluster view' than having
separate mechanisms. How the 'cross cluster' aspect of the discovery's
topology would be implemented in that case is yet another question, but I
suppose it could just as well use jms cross-cluster...

Cheers,
Stefan

>
>Just my 2 cents
>Dominik
>
>[0]http://markmail.org/thread/w3kgl7jxvhki3oqj
>
>
>On Tue, Apr 16, 2013 at 11:51 AM, Michael Dürig <md...@apache.org>
>wrote:
>
>>
>>
>> On 15.4.13 9:46, Julian Reschke wrote:
>>
>>> On 2013-04-15 10:32, Bertrand Delacretaz wrote:
>>>
>>
>>  So I'm wondering if using an existing distributed message queue
>>>> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
>>>> is only a problem in very large Oak setups, so having to install
>>>> additional components might not be an issue.
>>>>
>>>
>>> Could that also help with implementing proper JCR Locking (or are we
>>> there already???).
>>>
>>
>> Probably. The idea of making external coordinaters pluggable has come up
>> before: https://issues.apache.org/**jira/browse/OAK-150?**
>> focusedCommentId=13401328&**page=com.atlassian.jira.**
>> 
>>plugin.system.issuetabpanels:**comment-tabpanel#comment-**13401328<https:
>>//issues.apache.org/jira/browse/OAK-150?focusedCommentId=13401328&page=co
>>m.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13
>>401328>
>>
>> Michael
>>

Re: Scalability of JCR observation

[Dominik Süß <do...@gmail.com>]

I see some overlap with the latest work of Carsten in Sling regarding
Discovery API[0]. Since Sling typically should work uppon JCR / Oak it
might be good not to follow different patterns. For a combined solution I
do think it would be great to have one pluggable mediating system instead
of two which might have strange sideeffects for rejoin scenarios in a
cluster.

Just my 2 cents
Dominik

[0]http://markmail.org/thread/w3kgl7jxvhki3oqj


On Tue, Apr 16, 2013 at 11:51 AM, Michael Dürig <md...@apache.org> wrote:

>
>
> On 15.4.13 9:46, Julian Reschke wrote:
>
>> On 2013-04-15 10:32, Bertrand Delacretaz wrote:
>>
>
>  So I'm wondering if using an existing distributed message queue
>>> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
>>> is only a problem in very large Oak setups, so having to install
>>> additional components might not be an issue.
>>>
>>
>> Could that also help with implementing proper JCR Locking (or are we
>> there already???).
>>
>
> Probably. The idea of making external coordinaters pluggable has come up
> before: https://issues.apache.org/**jira/browse/OAK-150?**
> focusedCommentId=13401328&**page=com.atlassian.jira.**
> plugin.system.issuetabpanels:**comment-tabpanel#comment-**13401328<https://issues.apache.org/jira/browse/OAK-150?focusedCommentId=13401328&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13401328>
>
> Michael
>

Re: Scalability of JCR observation

[Michael Dürig <md...@apache.org>]

On 15.4.13 9:46, Julian Reschke wrote:
> On 2013-04-15 10:32, Bertrand Delacretaz wrote:

>> So I'm wondering if using an existing distributed message queue
>> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
>> is only a problem in very large Oak setups, so having to install
>> additional components might not be an issue.
>
> Could that also help with implementing proper JCR Locking (or are we
> there already???).

Probably. The idea of making external coordinaters pluggable has come up 
before: 
https://issues.apache.org/jira/browse/OAK-150?focusedCommentId=13401328&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13401328

Michael

Re: Scalability of JCR observation

[Thomas Mueller <mu...@adobe.com>]

Hi,

I think it would be a pity if we would have to add ActiveMQ/RabbitMQ to
Oak, as it would increase implementation and configuration complexity
quite a lot. I would really like to avoid that.

As for JCR locking, I believe the current cluster storage backend
(MongoMK) can quite easily support it, without having to add
ActiveMQ/RabbitMQ or any other direct communication between cluster nodes.

Regards,
Thomas


On 4/15/13 10:46 AM, "julian.reschke@gmx.de" <ju...@gmx.de> wrote:

>On 2013-04-15 10:32, Bertrand Delacretaz wrote:
>> Hi,
>>
>> On Fri, Apr 12, 2013 at 3:35 PM, Michael Dürig <md...@apache.org>
>>wrote:
>>> An implementation approach for backward compatible observation is to
>>>use a
>>> commit hook to record the required changes to a journal (e.g.
>>> /jcr:system/rep:observation). Observation listeners would then later
>>> generate the observation events by scrapping that journal
>>>asynchronously...
>>
>> This sounds a lot like a distributed message queue...
>>
>>> ...A somewhat open question is how this should work across the
>>>cluster...
>>
>> So I'm wondering if using an existing distributed message queue
>> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
>> is only a problem in very large Oak setups, so having to install
>> additional components might not be an issue.
>
>Could that also help with implementing proper JCR Locking (or are we
>there already???).
>
>Best regards, Julian
>

Re: Scalability of JCR observation

[Julian Reschke <ju...@gmx.de>]

On 2013-04-15 10:32, Bertrand Delacretaz wrote:
> Hi,
>
> On Fri, Apr 12, 2013 at 3:35 PM, Michael Dürig <md...@apache.org> wrote:
>> An implementation approach for backward compatible observation is to use a
>> commit hook to record the required changes to a journal (e.g.
>> /jcr:system/rep:observation). Observation listeners would then later
>> generate the observation events by scrapping that journal asynchronously...
>
> This sounds a lot like a distributed message queue...
>
>> ...A somewhat open question is how this should work across the cluster...
>
> So I'm wondering if using an existing distributed message queue
> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
> is only a problem in very large Oak setups, so having to install
> additional components might not be an issue.

Could that also help with implementing proper JCR Locking (or are we 
there already???).

Best regards, Julian

Re: Scalability of JCR observation

[Michael Dürig <md...@apache.org>]

On 15.4.13 9:32, Bertrand Delacretaz wrote:
> Hi,
>
> On Fri, Apr 12, 2013 at 3:35 PM, Michael Dürig <md...@apache.org> wrote:
>> An implementation approach for backward compatible observation is to use a
>> commit hook to record the required changes to a journal (e.g.
>> /jcr:system/rep:observation). Observation listeners would then later
>> generate the observation events by scrapping that journal asynchronously...
>
> This sounds a lot like a distributed message queue...

Right.

>
>> ...A somewhat open question is how this should work across the cluster...
>
> So I'm wondering if using an existing distributed message queue
> service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
> is only a problem in very large Oak setups, so having to install
> additional components might not be an issue.

Yes I think we can do that. In a first step I'd like to have an 
implementation entirely internal to Oak so we can validate it against 
our basic use cases. Further down the line we can abstract over the 
underlying queueing mechanism such that other backends could be pugged in.

Michael

>
> -Bertrand
>

Re: Scalability of JCR observation

[Bertrand Delacretaz <bd...@apache.org>]

Hi,

On Fri, Apr 12, 2013 at 3:35 PM, Michael Dürig <md...@apache.org> wrote:
> An implementation approach for backward compatible observation is to use a
> commit hook to record the required changes to a journal (e.g.
> /jcr:system/rep:observation). Observation listeners would then later
> generate the observation events by scrapping that journal asynchronously...

This sounds a lot like a distributed message queue...

> ...A somewhat open question is how this should work across the cluster...

So I'm wondering if using an existing distributed message queue
service (ActiveMQ/RabbitMQ etc) would help implement this. IIUC this
is only a problem in very large Oak setups, so having to install
additional components might not be an issue.

-Bertrand

Re: Scalability of JCR observation

[Julian Reschke <ju...@gmx.de>]

On 2013-04-12 15:35, Michael Dürig wrote:
> Hi,
>
> Currently JCR observation in Oak is targeted towards scalability and
> therefore compromises on some features:
>
> * Observation events are generated by diffing revisions. This results in
> a consolidated view where intermediated events might have cancelled out
> and the order of events is completely random.
>
> * No support Event.getUserId() and Event.getUserData().
> ...

* No PERSIST events, right?

Best regards, Julian

Re: Scalability of JCR observation

[Michael Dürig <md...@apache.org>]

See https://issues.apache.org/jira/browse/OAK-775

Michael

On 12.4.13 14:35, Michael Dürig wrote:
> Hi,
>
> Currently JCR observation in Oak is targeted towards scalability and
> therefore compromises on some features:
>
> * Observation events are generated by diffing revisions. This results in
> a consolidated view where intermediated events might have cancelled out
> and the order of events is completely random.
>
> * No support Event.getUserId() and Event.getUserData().
>
> Since this change might create backward compatibility issues for some
> applications, Angela, Marcel, Tom and I came up with the idea to
> implement a 2nd, more backward compatible approach to observation in
> Oak. This approach would in contrast to the former one come with
> compromises in scalability. Clients could choose between the
> implementations when registering an event listener (e.g using a marker
> interface).
>
> An implementation approach for backward compatible observation is to use
> a commit hook to record the required changes to a journal (e.g.
> /jcr:system/rep:observation). Observation listeners would then later
> generate the observation events by scrapping that journal asynchronously.
>
> Such an approach would increase the size of the repository by a constant
> factor < 2 since changes required for observation are also written to
> the journal. Furthermore it adds a somewhat expensive commit hook for
> recording such changes. The idea is that there is no such impact as long
> as backward compatible observation is not used.
>
> A somewhat open question is how this should work across the cluster. The
> best way is probably for cluster sync to exclude the observation journal
> and have the built by each cluster node individually.
>
> Michael