You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Carsten Ziegeler <cz...@apache.org> on 2015/06/15 09:43:09 UTC
Observation: External vs local - Load distribution
Hi,
with JR2 there was a clear distinction between local and external
events. A pattern that emerged out of this is to use the "local"
information of an observation event to start some processing. By using
this information you have a load distribution of the processing which is
equal to the distribution of changes - which kind of sounds nice.
Now, with Oak there is still this distinction, however if I remember
correctly under heavy load it might happen that local events are
reported as external events. And in that case the above pattern fails.
Regardless of how rare this situation might be, if it can happen it will
eventually happen.
I would like to have some way to distribute post processing of changes
while running on Oak in a similar way as with JR2 - I don't care that
much whether this uses local/external or anything else Oak provides. The
only guarantee I would need is that a single instance is handling the
changes. And this by simply using a mechanism or information Oak provides.
Is there anything I could use?
Regards
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 15.06.15 um 14:50 schrieb Stefan Egli:
> On 6/15/15 2:40 PM, "Carsten Ziegeler" <cz...@apache.org> wrote:
>
>> Am 15.06.15 um 14:23 schrieb Marcel Reutegger:
>>> Hi,
>>>
>>> you can write a CommitEditor, which is called with every
>>> local commit.
>>>
>>
>> Is it easy to calculate the changed nodes/properties in this editor?
>
> As I understand yes, the Editor gets callback for all changed nodes and
> properties.
>
>
> I guess the question is how that is encapsulated towards upper layers as
> you probably do not want (too much) application code using commit editors.
>
I assume, the CommitEditor only gets local events - right now, in
Sling's observation bridge we're registering an Observer, which gets
collapsed information, but from both local and external events.
Not sure, if this is a good idea, but we could replace that Observer
with a CommitEditor and get all information we need (including local and
which user changed it). We could then send out OSGi events based on that
information.
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 15.06.15 um 15:23 schrieb Julian Sedding:
> AFAIK the CommitEditor gets called before the commit is
> finalized/persisted. Later CommitEditors can still fail the commit.
> This may lead to events being reported, that did not actually happen.
>
> It needs to be considered if this represents a problem for the given use-case.
>
Oh...definitely, doesn't make sense to inform someone about something if
it's either not persisted at all or potentially persisted differently.
I guess there is nothing which I could use directly after it's committed
- which is not an Observer?
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Julian Sedding <js...@gmail.com>.
AFAIK the CommitEditor gets called before the commit is
finalized/persisted. Later CommitEditors can still fail the commit.
This may lead to events being reported, that did not actually happen.
It needs to be considered if this represents a problem for the given use-case.
Regards
Julian
On Mon, Jun 15, 2015 at 2:50 PM, Stefan Egli <st...@apache.org> wrote:
> On 6/15/15 2:40 PM, "Carsten Ziegeler" <cz...@apache.org> wrote:
>
>>Am 15.06.15 um 14:23 schrieb Marcel Reutegger:
>>> Hi,
>>>
>>> you can write a CommitEditor, which is called with every
>>> local commit.
>>>
>>
>>Is it easy to calculate the changed nodes/properties in this editor?
>
> As I understand yes, the Editor gets callback for all changed nodes and
> properties.
>
>
> I guess the question is how that is encapsulated towards upper layers as
> you probably do not want (too much) application code using commit editors.
>
> Cheers,
> Stefan
>
>
Re: Observation: External vs local - Load distribution
Posted by Stefan Egli <st...@apache.org>.
On 6/15/15 2:40 PM, "Carsten Ziegeler" <cz...@apache.org> wrote:
>Am 15.06.15 um 14:23 schrieb Marcel Reutegger:
>> Hi,
>>
>> you can write a CommitEditor, which is called with every
>> local commit.
>>
>
>Is it easy to calculate the changed nodes/properties in this editor?
As I understand yes, the Editor gets callback for all changed nodes and
properties.
I guess the question is how that is encapsulated towards upper layers as
you probably do not want (too much) application code using commit editors.
Cheers,
Stefan
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 15.06.15 um 14:23 schrieb Marcel Reutegger:
> Hi,
>
> you can write a CommitEditor, which is called with every
> local commit.
>
Is it easy to calculate the changed nodes/properties in this editor?
Carsten
> Regards
> Marcel
>
> On 15/06/15 09:43, "Carsten Ziegeler" wrote:
>
>> Hi,
>>
>> with JR2 there was a clear distinction between local and external
>> events. A pattern that emerged out of this is to use the "local"
>> information of an observation event to start some processing. By using
>> this information you have a load distribution of the processing which is
>> equal to the distribution of changes - which kind of sounds nice.
>>
>> Now, with Oak there is still this distinction, however if I remember
>> correctly under heavy load it might happen that local events are
>> reported as external events. And in that case the above pattern fails.
>> Regardless of how rare this situation might be, if it can happen it will
>> eventually happen.
>>
>> I would like to have some way to distribute post processing of changes
>> while running on Oak in a similar way as with JR2 - I don't care that
>> much whether this uses local/external or anything else Oak provides. The
>> only guarantee I would need is that a single instance is handling the
>> changes. And this by simply using a mechanism or information Oak provides.
>>
>> Is there anything I could use?
>>
>> Regards
>> Carsten
>> --
>> Carsten Ziegeler
>> Adobe Research Switzerland
>> cziegeler@apache.org
>
>
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Marcel Reutegger <mr...@adobe.com>.
Hi,
you can write a CommitEditor, which is called with every
local commit.
Regards
Marcel
On 15/06/15 09:43, "Carsten Ziegeler" wrote:
>Hi,
>
>with JR2 there was a clear distinction between local and external
>events. A pattern that emerged out of this is to use the "local"
>information of an observation event to start some processing. By using
>this information you have a load distribution of the processing which is
>equal to the distribution of changes - which kind of sounds nice.
>
>Now, with Oak there is still this distinction, however if I remember
>correctly under heavy load it might happen that local events are
>reported as external events. And in that case the above pattern fails.
>Regardless of how rare this situation might be, if it can happen it will
>eventually happen.
>
>I would like to have some way to distribute post processing of changes
>while running on Oak in a similar way as with JR2 - I don't care that
>much whether this uses local/external or anything else Oak provides. The
>only guarantee I would need is that a single instance is handling the
>changes. And this by simply using a mechanism or information Oak provides.
>
>Is there anything I could use?
>
>Regards
>Carsten
>--
>Carsten Ziegeler
>Adobe Research Switzerland
>cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Stefan Egli <st...@apache.org>.
Hi Carsten,
For external events the commit info is indeed not provided yup.
For internal ones it is - except for those 'overflow' ones which collapse
into a pseudo-external one.
Cheers,
Stefan
On 13/10/15 15:17, "Carsten Ziegeler" <cz...@apache.org> wrote:
>Am 17.06.15 um 10:35 schrieb Carsten Ziegeler:
>> Ok, just to recap. In Sling we can implement the Observer interface (and
>> not use the BackgroundObserver base class). This will give us reliably
>> user id for all local events.
>>
>> Does anyone see a problem with this approach?
>>
>Getting back to this problem, it seems the above does not work, as the
>DocumentNodeStore is not passing on the commit info to the observer in
>the case of external events.
>So no matter how I implement my observer, I don't get the info passed in.
>
>Can someone please confirm this?
>
>Thanks
>Carsten
>--
>Carsten Ziegeler
>Adobe Research Switzerland
>cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 17.06.15 um 10:35 schrieb Carsten Ziegeler:
> Ok, just to recap. In Sling we can implement the Observer interface (and
> not use the BackgroundObserver base class). This will give us reliably
> user id for all local events.
>
> Does anyone see a problem with this approach?
>
Getting back to this problem, it seems the above does not work, as the
DocumentNodeStore is not passing on the commit info to the observer in
the case of external events.
So no matter how I implement my observer, I don't get the info passed in.
Can someone please confirm this?
Thanks
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Michael Dürig <md...@apache.org>.
On 17.6.15 7:47 , Ian Boston wrote:
> Hi,
>
> +1 for unbounded, let the GC take care of it and log periodically when its
> size becomes significant so that anyone wondering why their JVM is
> consuming so much GC time gets a clue as to the cause, without having to
> perform heap dumps, thread dumps or jvm probes.
BackgroundObserver already allows you to specify the queue length [1].
Just set this to a very high value if you prefer to degrade via OOM
instead of collapsing local commits into external ones.
Michael
[1]
https://github.com/apache/jackrabbit-oak/blob/80985b51a18ef61015d263d2358f5c3713fea873/oak-core/src/main/java/org/apache/jackrabbit/oak/spi/commit/BackgroundObserver.java#L154
>
> (but ideally all queues would have simple, efficient, metrics that can be
> monitored all the time in production, not just by someone connecting a JMX
> console or Web Console)
>
> Best Regards
> Ian
>
> On 17 June 2015 at 09:56, Chetan Mehrotra <ch...@gmail.com> wrote:
>
>> Just ensure that your Observer is fast as its invoked the critical path.
>>
>> This would probably end up with a design similar to Background
>> Observer. May be better option would be to allow BO have non bounded
>> queue.
>> Chetan Mehrotra
>>
>>
>> On Wed, Jun 17, 2015 at 2:05 PM, Carsten Ziegeler <cz...@apache.org>
>> wrote:
>>> Ok, just to recap. In Sling we can implement the Observer interface (and
>>> not use the BackgroundObserver base class). This will give us reliably
>>> user id for all local events.
>>>
>>> Does anyone see a problem with this approach?
>>>
>>> Carsten
>>> --
>>> Carsten Ziegeler
>>> Adobe Research Switzerland
>>> cziegeler@apache.org
>>
>
Re: Observation: External vs local - Load distribution
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
+1 for unbounded, let the GC take care of it and log periodically when its
size becomes significant so that anyone wondering why their JVM is
consuming so much GC time gets a clue as to the cause, without having to
perform heap dumps, thread dumps or jvm probes.
(but ideally all queues would have simple, efficient, metrics that can be
monitored all the time in production, not just by someone connecting a JMX
console or Web Console)
Best Regards
Ian
On 17 June 2015 at 09:56, Chetan Mehrotra <ch...@gmail.com> wrote:
> Just ensure that your Observer is fast as its invoked the critical path.
>
> This would probably end up with a design similar to Background
> Observer. May be better option would be to allow BO have non bounded
> queue.
> Chetan Mehrotra
>
>
> On Wed, Jun 17, 2015 at 2:05 PM, Carsten Ziegeler <cz...@apache.org>
> wrote:
> > Ok, just to recap. In Sling we can implement the Observer interface (and
> > not use the BackgroundObserver base class). This will give us reliably
> > user id for all local events.
> >
> > Does anyone see a problem with this approach?
> >
> > Carsten
> > --
> > Carsten Ziegeler
> > Adobe Research Switzerland
> > cziegeler@apache.org
>
Re: Observation: External vs local - Load distribution
Posted by Chetan Mehrotra <ch...@gmail.com>.
Just ensure that your Observer is fast as its invoked the critical path.
This would probably end up with a design similar to Background
Observer. May be better option would be to allow BO have non bounded
queue.
Chetan Mehrotra
On Wed, Jun 17, 2015 at 2:05 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> Ok, just to recap. In Sling we can implement the Observer interface (and
> not use the BackgroundObserver base class). This will give us reliably
> user id for all local events.
>
> Does anyone see a problem with this approach?
>
> Carsten
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Ok, just to recap. In Sling we can implement the Observer interface (and
not use the BackgroundObserver base class). This will give us reliably
user id for all local events.
Does anyone see a problem with this approach?
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Michael Dürig <md...@apache.org>.
On 15.6.15 5:46 , Carsten Ziegeler wrote:
> And it would be great if that implementation could simply pass
> all filters down to the oak level
+1. Oak provides a rich and extensible interface for filtering
observation events [1].
org.apache.jackrabbit.oak.plugins.observation.filter.FilterBuilder might
be a good starting point.
Michael
[1] https://issues.apache.org/jira/browse/OAK-1133
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 15.06.15 um 17:31 schrieb Stefan Egli:
>
>>>
>> Ah, thanks Chetan, that's the first time I hear this - so basically if
>> we implement our own observer, we can reliably get:
>> a) all changes
>> b) local/external info
>> c) user id
>>
>> Is that correct?
>
> the way I understand it is: ;)
>
> * for local changes yes, you'd get all local changes incl user id
> * for external changes you'd get them all, but without user id and they
> would typically be collapsed (as external changes are only periodically
> written by the background updater)
Ah - that would be sufficient I think.
>
> So given this, you could indeed have an Observer that throws away all
> external events (which are easily spottable as they have commitInfo==null)
> and only process internal ones. And for such a 'local-only' observer I
> think this could be a feasible approach.
>
> Speaking more generally however: I guess to support scaling to very large
> number of instances, the goal should be that external events are filtered
> as much as possible too. Providing fast processing alone (as is the goal
> eg with OAK-2829) would not suffice. I think for this we'd need 'oak level
> observation filtering'. Such a filter could be applied to the journal
> (filling only 'interested' paths into the diff caches).
>
> At which point I wonder if it would not be beneficial to do both 'local vs
> external' as well as 'path-filtering' on an oak level, rather than one or
> both on the sling levelŠ
As you know, we're currently prototyping the new observation API in
Sling. And it would be great if that implementation could simply pass
all filters down to the oak level. Local/External is one, path is the
other one. Right now I don't see more.
>
> Re the commit editor use case: I think that would still be the only option
> if you'd want 'local-guaranteed' events, ie local events that would not
> get lost even in case of a crash. At the moment there are no solutions for
> this - local events just get lost. I think we could have three different
> event types (local-filtered, local-guaranteed-filtered, external-filtered).
Right, loosing events is a problem in the old situation (JR2), so I hope
we can ignore this. I guess the only way to guarantee something would
be a transaction as the post processing of the commit is potentially
using a different/new session. But if there is a chance to get to this
point, I'm not opposed to use it :)
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Stefan Egli <st...@apache.org>.
On 6/15/15 4:29 PM, "Carsten Ziegeler" <cz...@apache.org> wrote:
>Am 15.06.15 um 16:21 schrieb Chetan Mehrotra:
>> On Mon, Jun 15, 2015 at 1:13 PM, Carsten Ziegeler
>><cz...@apache.org> wrote:
>>> Now, with Oak there is still this distinction, however if I remember
>>> correctly under heavy load it might happen that local events are
>>> reported as external events. And in that case the above pattern fails.
>>> Regardless of how rare this situation might be, if it can happen it
>>>will
>>> eventually happen.
>>
>> This is an implementation detail of BackgroundObserver (BO) which is
>> used by OakResourceListener in Sling. BO keeps a queue of changed
>> NodeState tuples and if it gets filled it is collapsed. If you want to
>> avoid that at *any* cost that you can used a different impl which uses
>> say LinkedBlockingQueue and does not enforce any limit. That would be
>> similar to how JcrResourceListener works which uses an unbound in
>> memory queue
Indeed a good point!
>>
>Ah, thanks Chetan, that's the first time I hear this - so basically if
>we implement our own observer, we can reliably get:
>a) all changes
>b) local/external info
>c) user id
>
>Is that correct?
the way I understand it is: ;)
* for local changes yes, you'd get all local changes incl user id
* for external changes you'd get them all, but without user id and they
would typically be collapsed (as external changes are only periodically
written by the background updater)
So given this, you could indeed have an Observer that throws away all
external events (which are easily spottable as they have commitInfo==null)
and only process internal ones. And for such a 'local-only' observer I
think this could be a feasible approach.
Speaking more generally however: I guess to support scaling to very large
number of instances, the goal should be that external events are filtered
as much as possible too. Providing fast processing alone (as is the goal
eg with OAK-2829) would not suffice. I think for this we'd need 'oak level
observation filtering'. Such a filter could be applied to the journal
(filling only 'interested' paths into the diff caches).
At which point I wonder if it would not be beneficial to do both 'local vs
external' as well as 'path-filtering' on an oak level, rather than one or
both on the sling level
Re the commit editor use case: I think that would still be the only option
if you'd want 'local-guaranteed' events, ie local events that would not
get lost even in case of a crash. At the moment there are no solutions for
this - local events just get lost. I think we could have three different
event types (local-filtered, local-guaranteed-filtered, external-filtered).
Cheers,
Stefan
Re: Observation: External vs local - Load distribution
Posted by Carsten Ziegeler <cz...@apache.org>.
Am 15.06.15 um 16:21 schrieb Chetan Mehrotra:
> On Mon, Jun 15, 2015 at 1:13 PM, Carsten Ziegeler <cz...@apache.org> wrote:
>> Now, with Oak there is still this distinction, however if I remember
>> correctly under heavy load it might happen that local events are
>> reported as external events. And in that case the above pattern fails.
>> Regardless of how rare this situation might be, if it can happen it will
>> eventually happen.
>
> This is an implementation detail of BackgroundObserver (BO) which is
> used by OakResourceListener in Sling. BO keeps a queue of changed
> NodeState tuples and if it gets filled it is collapsed. If you want to
> avoid that at *any* cost that you can used a different impl which uses
> say LinkedBlockingQueue and does not enforce any limit. That would be
> similar to how JcrResourceListener works which uses an unbound in
> memory queue
>
Ah, thanks Chetan, that's the first time I hear this - so basically if
we implement our own observer, we can reliably get:
a) all changes
b) local/external info
c) user id
Is that correct?
Regards
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
cziegeler@apache.org
Re: Observation: External vs local - Load distribution
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Mon, Jun 15, 2015 at 1:13 PM, Carsten Ziegeler <cz...@apache.org> wrote:
> Now, with Oak there is still this distinction, however if I remember
> correctly under heavy load it might happen that local events are
> reported as external events. And in that case the above pattern fails.
> Regardless of how rare this situation might be, if it can happen it will
> eventually happen.
This is an implementation detail of BackgroundObserver (BO) which is
used by OakResourceListener in Sling. BO keeps a queue of changed
NodeState tuples and if it gets filled it is collapsed. If you want to
avoid that at *any* cost that you can used a different impl which uses
say LinkedBlockingQueue and does not enforce any limit. That would be
similar to how JcrResourceListener works which uses an unbound in
memory queue
Chetan Mehrotra