You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Alexander Klimetschek <ak...@adobe.com> on 2013/11/07 02:49:26 UTC
Observation
Hi everyone,
having looked at and discussed observation recently (partly on our Adobe internal list with some of our application use cases, partly with Toby f2f), here are some things that are IMHO worthwile to look at. I hope I can contribute a bit, but before I get pulled away again, I wanted at least summarize and persist() my ideas:
1) listeners should not get external events by default
OAK-1121 helps, but applications must say they exclude external events, while it should be the other way around (only local by default) - this can arguably not change because of the jcr observation contract. The current implementation can still be optimized a bit, see my comment in the issue.
2) filtering happens too late within listeners because jcr observation API is too simple (OAK-1133, "observation listener plus")
The filter logic would be passed declaratively and allow more options as noted in OAK-1133:
addEventListener(listener, Filter.Path("/libs"), Filter.PropertyValue("jcr:content/sling:resourceType", "myValue"), ...)
(just a rough sketch; the simplest solution could simply introduce a FilterEventListener interface that extends the existing EventListener and provides the filter in a getFilter() method; a simple instanceof check within oak-jcr could detect that; but a completely separate API might also make sense, see point 6 below)
This could then be evaluated when the NodeStateDiff is calculated. What is good now already is that the base path (if specified in the jcr observation) is evaluated as soon as getting the diff roots. But many real use cases might want to have multiple paths (/apps OR /libs) or globbing (/content/site/*/jcr:content) - this could be easily and efficiently added here in one place (amongst those other filters). Having one or few base paths is in the oak case probably the best way to filter things and one should avoid listeners that register at root, and only filter on node types for example, which requires a lot more tree walking.
In our app (CQ) we have 18 listeners that register for all events, because the JCR observation doesn't give them the right filter options, so they have to do it themselves. I expect them to be vastly improved with this new approach.
Also, getting events for deleted nodes could be supported here, since there is full access to the old NodeState.
The ideal solution could also be built backwards compatible: implement the same filtering with normal JCR API usage and automatically wrap the listener in another event listener that filters the "manual" way. This allows application code that works with both Jackrabbit 2 or Oak with minimal effort (would need some helper in jackrabbit-jcr-commons or so). Unless we want to backport this to Jackrabbit 2 as well.
3) cluster events
As agreed, external events can be the real scalability issue. It was also noted, that many cases do not need it at all (hence 1). But there are still cases that need it: in Sling for example, code deployment (bundles, JSP scripts, etc.) is based on the JCR repository, meaning when you put a bundle in a certain "install" folder in the JCR, all cluster instances need to pick it up. However, such deployment events are comparably rare (except on a developer machine :)), so the throughput doesn't have to be high. OTOH large-scale content changes will (and should) be handled locally, thus not requiring external propagation.
Now the question is if 2) with a filter-as-detailed-and-early-as-possible approach solves this already (having the right external/non-external flag on all listeners, plus detailed filters for the external listeners), or if we could make use of the fairly rare nature of those events and try to filter on the _source_ instance already and propagate those few events in some other, efficient way to other cluster instances. This might improve performance as the ChangeDispatcher#externalChange() check (polling all 100ms) could then be removed completely. But if the filtering on the NodeStateDiff is fast (because listeners are specific), then maybe it is ok.
4) listener threads
Currently in oak each event listener gets its own thread. In our app CQ we have about 150 listeners, so you end up introducing 150 threads. I am not sure if this is a good idea.
Jackrabbit 2 had a single thread, which had the issue of one observation listener blocking all others.
An obvious solution would be a thread pool (probably configurable number of threads). And on top of that, if the pool is full and no thread is free anymore, one could simply kill blocked handlers after a certain timeout (and optionally blacklist them).
5) filtering in central thread?
There was some discussion about the current filtering evaluation [0]. Each listener has its own ChangeProcessor which in turn has its own ChangeDispatcher.Listener. This means filtering is not in a single, central thread, but happens in each listener's thread (they each have their own ChangeSet queue, which indeed might be different because the previous root NodeState to compare to might be different for each).
I thought this was not optimal but it seems I was wrong - splitting that up and generating events in a central thread and then passing them in a queue to the listeners actually turned out to scale worse (in a quick test with many parallel listeners). See my patch at [1].
I guess it is that reading asynchronously from the immutable NodeStates is more efficient than multiple blocking queues. Which speaks good for the underlying oak implementation :)
[0] http://markmail.org/thread/533orsfr44wllvrx
[1] https://github.com/alexkli/jackrabbit-oak/commit/aee631ded4996194a3ad0dec1fc7a9917f7123b8
6) long-running session just for observation
One problem with JCR observation is that you need one session open all the time "just for the observation". This is done to be able to run observation under the permissions of a certain user. But nobody in practice uses the session for anything else than reading data upon events (which mostly is done to filter only, see 2); when you need to write things in an event, best practice is to create a new session on demand an close it again. With Oak's refresh() policy this is even more important.
Maybe a new API from 2) could skip the requirement of an open session. Registration would be based upon a session to handle the credentials easily, but that session could be closed without killing the listener. You would get back a special listener object that would need to be kept alive and referenced to be able to clean up listeners automatically on a finalize() if they forget to unregister.
Or you keep the session, but mark it in a special way so that Oak can internally save resources. Or maybe the session in oak is already so light-weight that this is actually no problem at all. I am just mentioning this because of the many warnings I get from oak in the logs about sessions that are open very long (and don't call refresh() IIRC).
7) move code to oak-jcr (minor)
Quite a bit of the code in oak.plugins.observation is currently JCR eventing specific (EventGenerator for example), afaics this is better suited in oak-jcr. Although with 2) it might change a bit anyway.
Cheers,
Alex
Re: Observation
Posted by Alexander Klimetschek <ak...@adobe.com>.
On 11.11.2013, at 10:29, Jukka Zitting <ju...@gmail.com> wrote:
> So instead of the spec, I think we should look at existing client code
> to determine what kind of expectations they have.
A quick search across our large application code base (Apache Sling & Adobe CQ) doesn't yield a single EventJournal usage.
Cheers,
Alex
Re: Observation
Posted by Ate Douma <at...@douma.nu>.
On 11/21/2013 07:18 PM, Jukka Zitting wrote:
> Hi,
>
> On Tue, Nov 19, 2013 at 9:47 AM, Ate Douma <at...@douma.nu> wrote:
>> On 11/11/2013 07:29 PM, Jukka Zitting wrote:
>>> Of course section 12.6.2 contradicts that by essentially giving the
>>> repository free reign to decide which (if any) events to include in
>>> the journal.
>>>
>> Far more problematic (broken really) is the fact that the spec. uses Dates
>> to access/order (skipTo) the events. Which is unreliable and useless in a
>> clustered environment. Meaning: unreliable for any serious use-case.
>
> Right. It only works for cases where the client won't mind seeing
> duplicate events at skip boundaries (and has a clock that's in sync
> with that of the repository).
>
>> I'm not following the current state of things at Oak enough to know if it
>> also will provide some kind of guaranteed orderable and cluster-wide
>> transaction revision/timestamp/whatever.
>
> We indeed do have that. It's possible for a client to "checkpoint" a
> repository at a specific revision and later get the exact set of
> changes between that checkpoint and any later state of the repository.
> For example the asynchronous index update task uses this feature to
> periodically update the index with changes that occurred between two
> checkpoints.
Good to hear, then we should be OK porting our functionality to Oak when that
time comes :)
Thanks for the heads-up!
Ate
>
> BR,
>
> Jukka Zitting
>
Re: Observation
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Tue, Nov 19, 2013 at 9:47 AM, Ate Douma <at...@douma.nu> wrote:
> On 11/11/2013 07:29 PM, Jukka Zitting wrote:
>> Of course section 12.6.2 contradicts that by essentially giving the
>> repository free reign to decide which (if any) events to include in
>> the journal.
>>
> Far more problematic (broken really) is the fact that the spec. uses Dates
> to access/order (skipTo) the events. Which is unreliable and useless in a
> clustered environment. Meaning: unreliable for any serious use-case.
Right. It only works for cases where the client won't mind seeing
duplicate events at skip boundaries (and has a clock that's in sync
with that of the repository).
> I'm not following the current state of things at Oak enough to know if it
> also will provide some kind of guaranteed orderable and cluster-wide
> transaction revision/timestamp/whatever.
We indeed do have that. It's possible for a client to "checkpoint" a
repository at a specific revision and later get the exact set of
changes between that checkpoint and any later state of the repository.
For example the asynchronous index update task uses this feature to
periodically update the index with changes that occurred between two
checkpoints.
BR,
Jukka Zitting
Re: Observation
Posted by Ate Douma <at...@douma.nu>.
On 11/11/2013 07:29 PM, Jukka Zitting wrote:
> Hi,
>
> On Mon, Nov 11, 2013 at 6:10 AM, Michael Dürig <md...@apache.org> wrote:
>> On 8.11.13 10:37 , Alexander Klimetschek wrote:
>>> Regarding journaled observation: retrieving a journal with
>>> ObservationManager#getEventJournal can allow access to events that
>>> happend earlier, right?
>>
>> The spec. doesn't say much here. I'd interpret it as getEventJournal()
>> returns the journal from the point where that call was made rather then from
>> the beginning of times.
>
> This interpretation makes the event journal essentially useless, as
> one could achieve the same functionality with a normal observation
> listener.
>
> The point of journaled observation is to be able to access past
> events, which the spec confirms (section 12.6):
>
> "Journaled observation allows an application to periodically connect
> to the repository and receive a report of changes that have occurred
> since some specified point in the past (for example, since the last
> connection)."
>
> Of course section 12.6.2 contradicts that by essentially giving the
> repository free reign to decide which (if any) events to include in
> the journal.
>
Far more problematic (broken really) is the fact that the spec. uses Dates to
access/order (skipTo) the events. Which is unreliable and useless in a clustered
environment. Meaning: unreliable for any serious use-case.
Luckily, Jackrabbit internally maintains a properly ordered and cluster-wide
journal item revision, as needed anyway for its cluster synchronization.
At Hippo we expose and use this feature through an extended RevisionEventJournal
[1] adding a skipToRevision(long) method and an RevisionEvent [2] exposing the
current event its revision.
Implementing this on top of Jackrabbit was really trivial [3] and enables us to
reliably and asynchronously process past events, in proper order, for things
like cross-cluster synchronization/replication, etc.
I'm not following the current state of things at Oak enough to know if it also
will provide some kind of guaranteed orderable and cluster-wide transaction
revision/timestamp/whatever. But it definitely would be worthwhile (and
important to us) to be available. Through a fixed/extended EventJournal
interface like we implemented on top of Jackrabbit or something else doesn't
really matter, but the functionality IMO is important enough to take into
consideration for Oak.
Regards, Ate
[1]
http://svn.onehippo.org/repos/hippo/hippo-cms7/repository/trunk/api/src/main/java/org/hippoecm/repository/api/RevisionEventJournal.java
[2]
http://svn.onehippo.org/repos/hippo/hippo-cms7/repository/trunk/api/src/main/java/org/hippoecm/repository/api/RevisionEvent.java
[3]
http://svn.onehippo.org/repos/hippo/hippo-cms7/repository/trunk/engine/src/main/java/org/apache/jackrabbit/core/observation/RevisionEventJournalImpl.java
> So instead of the spec, I think we should look at existing client code
> to determine what kind of expectations they have.
>
> BR,
>
> Jukka Zitting
>
Re: Observation
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Mon, Nov 11, 2013 at 6:10 AM, Michael Dürig <md...@apache.org> wrote:
> On 8.11.13 10:37 , Alexander Klimetschek wrote:
>> Regarding journaled observation: retrieving a journal with
>> ObservationManager#getEventJournal can allow access to events that
>> happend earlier, right?
>
> The spec. doesn't say much here. I'd interpret it as getEventJournal()
> returns the journal from the point where that call was made rather then from
> the beginning of times.
This interpretation makes the event journal essentially useless, as
one could achieve the same functionality with a normal observation
listener.
The point of journaled observation is to be able to access past
events, which the spec confirms (section 12.6):
"Journaled observation allows an application to periodically connect
to the repository and receive a report of changes that have occurred
since some specified point in the past (for example, since the last
connection)."
Of course section 12.6.2 contradicts that by essentially giving the
repository free reign to decide which (if any) events to include in
the journal.
So instead of the spec, I think we should look at existing client code
to determine what kind of expectations they have.
BR,
Jukka Zitting
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
On 8.11.13 10:37 , Alexander Klimetschek wrote:
> Regarding journaled observation: retrieving a journal with
> ObservationManager#getEventJournal can allow access to events that
> happend earlier, right?
The spec. doesn't say much here. I'd interpret it as getEventJournal()
returns the journal from the point where that call was made rather then
from the beginning of times.
So the goal of supporting filtering as early
> as possible wouldn't quite work here, as you still have to keep
> access to all events for anyone asking for them later. OTOH, the spec
> seems to leave journal implementations a lot of freedom on how many
> events are still accessible at all (which makes me wonder how much
> you can make application code rely on it) [0].
The same actually holds true for asynchronous observation:
http://www.day.com/specs/jcr/2.0/12_Observation.html#12.2%20Scope%20of%20Event%20Reporting
>
> I assume due to the MVCC principle in oak it is easy to implement the
> journal and it is cheap to "keep all events". How far can you go
> back? I assume that depends on the implementation, e.g. when the last
> tar optimization happened?
Right, just diff old revisions as needed. The retention time for old
revisions is currently an implementation detail. But see OAK-114 for an
ongoing effort on this.
Michael
Re: Observation
Posted by Alexander Klimetschek <ak...@adobe.com>.
On 08.11.2013, at 03:45, Michael Dürig <md...@apache.org> wrote:
> Just occurred to me that in JCR 2.1 the current way of specifying a
> filter is deprecated in favour of a dedicated EventFilter class. I think
> this sketches a way forward here.
Interesting! Albeit it just is a simple data class to contain all the different parameters that make the current ObservationManager#addEventListener a bit ugly. And you are supposed to have the repository implementation create it for you using ObservationManager#createEventFilter. But at least it is an interface to work with (aka to extend from) and it is the same for both normal and journaled observation.
Regarding journaled observation: retrieving a journal with ObservationManager#getEventJournal can allow access to events that happend earlier, right? So the goal of supporting filtering as early as possible wouldn't quite work here, as you still have to keep access to all events for anyone asking for them later. OTOH, the spec seems to leave journal implementations a lot of freedom on how many events are still accessible at all (which makes me wonder how much you can make application code rely on it) [0].
I assume due to the MVCC principle in oak it is easy to implement the journal and it is cheap to "keep all events". How far can you go back? I assume that depends on the implementation, e.g. when the last tar optimization happened?
[0] http://www.day.com/specs/jcr/2.0/12_Observation.html#12.6.2%20Journaling%20Configuration
Cheers,
Alex
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
On 8.11.13 11:28 , Michael Dürig wrote:
>> (just a rough sketch; the simplest solution could simply introduce a
>> FilterEventListener interface that extends the existing EventListener
>> and provides the filter in a getFilter() method; a simple instanceof
>> check within oak-jcr could detect that; but a completely separate API
>> might also make sense, see point 6 below)
>
> This wouldn't work for journalled observation (OAK-1145) where we'd also
> need this filtering capabilities. There isn't an event listener involved
> there where to attach such a filter. So we probably need that new API
> where some kind of a filter object is passed directly.
Just occurred to me that in JCR 2.1 the current way of specifying a
filter is deprecated in favour of a dedicated EventFilter class. I think
this sketches a way forward here.
Michael
Re: Observation
Posted by Alexander Klimetschek <ak...@adobe.com>.
On 08.11.2013, at 02:28, Michael Dürig <md...@apache.org> wrote:
> This wouldn't work for journalled observation (OAK-1145) where we'd also
> need this filtering capabilities. There isn't an event listener involved
> there where to attach such a filter. So we probably need that new API
> where some kind of a filter object is passed directly.
Oh, yes, I did not look at journalled observation, probably because I haven't used it on the app layer yet ;-) It is rarely used, but I expect a few use cases, so we should think about a similar API here as well (using the same filter definitions to make it seemless).
> On the implementation side we will need a more general, flexible and
> composeable filter mechanism than we currently have. Due to the low
> level nature of these filters (working on the node state diff), these
> filters will most likely be somewhat awkward to use directly. On the API
> side we should therefore provide a convenient way to compose and inject
> such filters. Let's use OAK-1133 to follow up on this.
Yes, that's why I think it should be a filter definition and the implementation can then adapt to. While it would be nice if you could have the listener pass a generic "boolean filter(Node, ...)" method, this would have to be tied to the JCR API or some oak internals like NodeState, which would be less optimal. I see the new API more as an extension to JCR rather than something specific to the current Oak.
> This somewhat interact with what we've done on OAK-803: the listening
> session gets refreshed before observation events are delivered. This was
> added to make observation more backward compatible. If we add a way to
> explicitly disable this refreshing the listening session will have
> access to deleted nodes. Added nodes will then not be visible until an
> explicit refresh is done.
Great! Then we can actually do both independently: a) support filtering on deleted nodes (which is what most of the access to deleted nodes in listeners is used for afaics, for the actual work they might just want to pick up the path of the deleted node and update some other place/cache). And b) read old/new data in the listener, based on calling refresh() or not.
> I think there are three scenarios for applications:
> 1. Only rely on cluster local events if possible,
> 2. otherwise apply a sufficiently specific filter to not to get swamped,
> 3. come up with a custom solution (i.e. MQ based on an Observer).
Ack.
> Separate threads where introduced with OAK-1113 to expedite event
> delivery, which also to increased compatibility with Sling and solved
> OAK-1084.
> I think we can further improve this by introducing a thread pool. When
> integrating this with the Whiteboard we would put the deployment into
> control on weighting observation throughput against resource usage. I
> wouldn't go so far and kill blocked thread for the same reason Jukka
> mentioned in his reply. However giving the deployment control over the
> thread pool makes it possible to use an unbounded pool that starts out
> with a few threads. Blocking handlers would then lead to higher resource
> usage while events are still being delivered to non blocked listeners.
Sounds good.
>> I guess it is that reading asynchronously from the immutable
>> NodeStates is more efficient than multiple blocking queues. Which
>> speaks good for the underlying oak implementation :)
>
> Good to have someone comment who actually got his hands dirty before
> being smart ;-)
Yes, I totally forgot about the positive effect of threads when you have 8 cpu cores :)
> I wouldn't care about the extra sessions. These should be cheap. And if
> they aren't we should fix that.
Good. Having something else than the session as basis would actually lead to unnecessary API and lifecycle management complexity.
> Re. the warning: this should only be
> logged once per session and is meant as help for migrating to Oak. We
> can try to further lower the noise if required or remove it entirely.
Maybe those messages can be put into a separate log (in default deployments). Because they include the stack trace (for a good reason) and occur especially when something is happening, they make looking at the error.log during debugging an application a bit harder.
> Many parts just recently moved to oak-core (OAK-950). After untangling
> the dependencies (OAK-1143, ongoing) it might be possible to move the
> respective parts back to oak-jcr.
I see.
Cheers,
Alex
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
Hi Alex,
Thanks for the detailed analysis. I generally agree. See inline for some
further comments.
On 7.11.13 2:49 , Alexander Klimetschek wrote:
> 1) listeners should not get external events by default
>
> OAK-1121 helps, but applications must say they exclude external
> events, while it should be the other way around (only local by
> default) - this can arguably not change because of the jcr
> observation contract. The current implementation can still be
> optimized a bit, see my comment in the issue.
Ack. I think we shouldn't/can't change this ATM due to the API contract
and backwards compatibility. Rather should applications facing issues
with being flooded with observation events redesign their use of
observation in a way that scales with the cluster deployment. Oak should
of course provide as much as possible to ease this process (e.g.
OAK-1133, OAK-1121).
>
> 2) filtering happens too late within listeners because jcr
> observation API is too simple (OAK-1133, "observation listener
> plus")
>
> The filter logic would be passed declaratively and allow more options
> as noted in OAK-1133:
>
> addEventListener(listener, Filter.Path("/libs"),
> Filter.PropertyValue("jcr:content/sling:resourceType", "myValue"),
> ...)
>
> (just a rough sketch; the simplest solution could simply introduce a
> FilterEventListener interface that extends the existing EventListener
> and provides the filter in a getFilter() method; a simple instanceof
> check within oak-jcr could detect that; but a completely separate API
> might also make sense, see point 6 below)
This wouldn't work for journalled observation (OAK-1145) where we'd also
need this filtering capabilities. There isn't an event listener involved
there where to attach such a filter. So we probably need that new API
where some kind of a filter object is passed directly.
>
> This could then be evaluated when the NodeStateDiff is calculated.
> What is good now already is that the base path (if specified in the
> jcr observation) is evaluated as soon as getting the diff roots. But
> many real use cases might want to have multiple paths (/apps OR
> /libs) or globbing (/content/site/*/jcr:content) - this could be
> easily and efficiently added here in one place (amongst those other
> filters). Having one or few base paths is in the oak case probably
> the best way to filter things and one should avoid listeners that
> register at root, and only filter on node types for example, which
> requires a lot more tree walking.
On the implementation side we will need a more general, flexible and
composeable filter mechanism than we currently have. Due to the low
level nature of these filters (working on the node state diff), these
filters will most likely be somewhat awkward to use directly. On the API
side we should therefore provide a convenient way to compose and inject
such filters. Let's use OAK-1133 to follow up on this.
>
> In our app (CQ) we have 18 listeners that register for all events,
> because the JCR observation doesn't give them the right filter
> options, so they have to do it themselves. I expect them to be vastly
> improved with this new approach.
>
> Also, getting events for deleted nodes could be supported here, since
> there is full access to the old NodeState.
This somewhat interact with what we've done on OAK-803: the listening
session gets refreshed before observation events are delivered. This was
added to make observation more backward compatible. If we add a way to
explicitly disable this refreshing the listening session will have
access to deleted nodes. Added nodes will then not be visible until an
explicit refresh is done.
>
> The ideal solution could also be built backwards compatible:
> implement the same filtering with normal JCR API usage and
> automatically wrap the listener in another event listener that
> filters the "manual" way. This allows application code that works
> with both Jackrabbit 2 or Oak with minimal effort (would need some
> helper in jackrabbit-jcr-commons or so). Unless we want to backport
> this to Jackrabbit 2 as well.
>
> 3) cluster events
>
> As agreed, external events can be the real scalability issue. It was
> also noted, that many cases do not need it at all (hence 1). But
> there are still cases that need it: in Sling for example, code
> deployment (bundles, JSP scripts, etc.) is based on the JCR
> repository, meaning when you put a bundle in a certain "install"
> folder in the JCR, all cluster instances need to pick it up. However,
> such deployment events are comparably rare (except on a developer
> machine :)), so the throughput doesn't have to be high. OTOH
> large-scale content changes will (and should) be handled locally,
> thus not requiring external propagation.
I think there are three scenarios for applications:
1. Only rely on cluster local events if possible,
2. otherwise apply a sufficiently specific filter to not to get swamped,
3. come up with a custom solution (i.e. MQ based on an Observer).
>
> Now the question is if 2) with a
> filter-as-detailed-and-early-as-possible approach solves this already
> (having the right external/non-external flag on all listeners, plus
> detailed filters for the external listeners), or if we could make use
> of the fairly rare nature of those events and try to filter on the
> _source_ instance already and propagate those few events in some
> other, efficient way to other cluster instances. This might improve
> performance as the ChangeDispatcher#externalChange() check (polling
> all 100ms) could then be removed completely. But if the filtering on
> the NodeStateDiff is fast (because listeners are specific), then
> maybe it is ok.
>
> 4) listener threads
>
> Currently in oak each event listener gets its own thread. In our app
> CQ we have about 150 listeners, so you end up introducing 150
> threads. I am not sure if this is a good idea.
>
> Jackrabbit 2 had a single thread, which had the issue of one
> observation listener blocking all others.
>
> An obvious solution would be a thread pool (probably configurable
> number of threads). And on top of that, if the pool is full and no
> thread is free anymore, one could simply kill blocked handlers after
> a certain timeout (and optionally blacklist them).
Separate threads where introduced with OAK-1113 to expedite event
delivery, which also to increased compatibility with Sling and solved
OAK-1084.
I think we can further improve this by introducing a thread pool. When
integrating this with the Whiteboard we would put the deployment into
control on weighting observation throughput against resource usage. I
wouldn't go so far and kill blocked thread for the same reason Jukka
mentioned in his reply. However giving the deployment control over the
thread pool makes it possible to use an unbounded pool that starts out
with a few threads. Blocking handlers would then lead to higher resource
usage while events are still being delivered to non blocked listeners.
>
> 5) filtering in central thread?
>
> There was some discussion about the current filtering evaluation [0].
> Each listener has its own ChangeProcessor which in turn has its own
> ChangeDispatcher.Listener. This means filtering is not in a single,
> central thread, but happens in each listener's thread (they each have
> their own ChangeSet queue, which indeed might be different because
> the previous root NodeState to compare to might be different for
> each).
>
> I thought this was not optimal but it seems I was wrong - splitting
> that up and generating events in a central thread and then passing
> them in a queue to the listeners actually turned out to scale worse
> (in a quick test with many parallel listeners). See my patch at [1].
>
> I guess it is that reading asynchronously from the immutable
> NodeStates is more efficient than multiple blocking queues. Which
> speaks good for the underlying oak implementation :)
>
> [0] http://markmail.org/thread/533orsfr44wllvrx [1]
> https://github.com/alexkli/jackrabbit-oak/commit/aee631ded4996194a3ad0dec1fc7a9917f7123b8
Good to have someone comment who actually got his hands dirty before
being smart ;-)
>
> 6) long-running session just for observation
>
> One problem with JCR observation is that you need one session open
> all the time "just for the observation". This is done to be able to
> run observation under the permissions of a certain user. But nobody
> in practice uses the session for anything else than reading data upon
> events (which mostly is done to filter only, see 2); when you need to
> write things in an event, best practice is to create a new session on
> demand an close it again. With Oak's refresh() policy this is even
> more important.
>
> Maybe a new API from 2) could skip the requirement of an open
> session. Registration would be based upon a session to handle the
> credentials easily, but that session could be closed without killing
> the listener. You would get back a special listener object that would
> need to be kept alive and referenced to be able to clean up listeners
> automatically on a finalize() if they forget to unregister.
>
> Or you keep the session, but mark it in a special way so that Oak can
> internally save resources. Or maybe the session in oak is already so
> light-weight that this is actually no problem at all. I am just
> mentioning this because of the many warnings I get from oak in the
> logs about sessions that are open very long (and don't call refresh()
> IIRC).
I wouldn't care about the extra sessions. These should be cheap. And if
they aren't we should fix that. Re. the warning: this should only be
logged once per session and is meant as help for migrating to Oak. We
can try to further lower the noise if required or remove it entirely.
>
> 7) move code to oak-jcr (minor)
>
> Quite a bit of the code in oak.plugins.observation is currently JCR
> eventing specific (EventGenerator for example), afaics this is better
> suited in oak-jcr. Although with 2) it might change a bit anyway.
Many parts just recently moved to oak-core (OAK-950). After untangling
the dependencies (OAK-1143, ongoing) it might be possible to move the
respective parts back to oak-jcr.
Michael
>
> Cheers, Alex
>
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
On 7.11.13 4:03 , Felix Meschberger wrote:
> For the new mechanism and distribution: May I suggest to explore a
> different route and use the OSGi EventAdmin service ? Yes, I know
> this would bake leveraging this new mechanism into OSGi and it would
> force to think about how event filtering takes place. But it would
> allow for reuse of good, proven and scalable event distribution
> system (since OSGi EventAdmin service implementations already have to
> implement synchronous and asynchronous multi-threaded delivery with
> optional blacklisting of long-running listeners allong with event
> serialization). If done in a way that would be compatible with
> Sling’s JCR Listener (JCR to OSGi Event Forwarding), the latter could
> even be replaced ...
I think this should be possible through an observer re-using the low
level filtering capabilities we anyway need for OAK-1133. When the low
level stuff for OAK-1133 is in place, we could even start drafting this
in Sling thus challenging Oak's modularity a bit.
Michael
Re: Observation
Posted by Felix Meschberger <fm...@adobe.com>.
Hi
I like this writeup. Thanks.
For the new mechanism and distribution: May I suggest to explore a different route and use the OSGi EventAdmin service ? Yes, I know this would bake leveraging this new mechanism into OSGi and it would force to think about how event filtering takes place. But it would allow for reuse of good, proven and scalable event distribution system (since OSGi EventAdmin service implementations already have to implement synchronous and asynchronous multi-threaded delivery with optional blacklisting of long-running listeners allong with event serialization). If done in a way that would be compatible with Sling’s JCR Listener (JCR to OSGi Event Forwarding), the latter could even be replaced ...
Regards
Felix
Am 07.11.2013 um 02:49 schrieb Alexander Klimetschek <ak...@adobe.com>:
> Hi everyone,
>
> having looked at and discussed observation recently (partly on our Adobe internal list with some of our application use cases, partly with Toby f2f), here are some things that are IMHO worthwile to look at. I hope I can contribute a bit, but before I get pulled away again, I wanted at least summarize and persist() my ideas:
>
> 1) listeners should not get external events by default
>
> OAK-1121 helps, but applications must say they exclude external events, while it should be the other way around (only local by default) - this can arguably not change because of the jcr observation contract. The current implementation can still be optimized a bit, see my comment in the issue.
>
> 2) filtering happens too late within listeners because jcr observation API is too simple (OAK-1133, "observation listener plus")
>
> The filter logic would be passed declaratively and allow more options as noted in OAK-1133:
>
> addEventListener(listener, Filter.Path("/libs"), Filter.PropertyValue("jcr:content/sling:resourceType", "myValue"), ...)
>
> (just a rough sketch; the simplest solution could simply introduce a FilterEventListener interface that extends the existing EventListener and provides the filter in a getFilter() method; a simple instanceof check within oak-jcr could detect that; but a completely separate API might also make sense, see point 6 below)
>
> This could then be evaluated when the NodeStateDiff is calculated. What is good now already is that the base path (if specified in the jcr observation) is evaluated as soon as getting the diff roots. But many real use cases might want to have multiple paths (/apps OR /libs) or globbing (/content/site/*/jcr:content) - this could be easily and efficiently added here in one place (amongst those other filters). Having one or few base paths is in the oak case probably the best way to filter things and one should avoid listeners that register at root, and only filter on node types for example, which requires a lot more tree walking.
>
> In our app (CQ) we have 18 listeners that register for all events, because the JCR observation doesn't give them the right filter options, so they have to do it themselves. I expect them to be vastly improved with this new approach.
>
> Also, getting events for deleted nodes could be supported here, since there is full access to the old NodeState.
>
> The ideal solution could also be built backwards compatible: implement the same filtering with normal JCR API usage and automatically wrap the listener in another event listener that filters the "manual" way. This allows application code that works with both Jackrabbit 2 or Oak with minimal effort (would need some helper in jackrabbit-jcr-commons or so). Unless we want to backport this to Jackrabbit 2 as well.
>
> 3) cluster events
>
> As agreed, external events can be the real scalability issue. It was also noted, that many cases do not need it at all (hence 1). But there are still cases that need it: in Sling for example, code deployment (bundles, JSP scripts, etc.) is based on the JCR repository, meaning when you put a bundle in a certain "install" folder in the JCR, all cluster instances need to pick it up. However, such deployment events are comparably rare (except on a developer machine :)), so the throughput doesn't have to be high. OTOH large-scale content changes will (and should) be handled locally, thus not requiring external propagation.
>
> Now the question is if 2) with a filter-as-detailed-and-early-as-possible approach solves this already (having the right external/non-external flag on all listeners, plus detailed filters for the external listeners), or if we could make use of the fairly rare nature of those events and try to filter on the _source_ instance already and propagate those few events in some other, efficient way to other cluster instances. This might improve performance as the ChangeDispatcher#externalChange() check (polling all 100ms) could then be removed completely. But if the filtering on the NodeStateDiff is fast (because listeners are specific), then maybe it is ok.
>
> 4) listener threads
>
> Currently in oak each event listener gets its own thread. In our app CQ we have about 150 listeners, so you end up introducing 150 threads. I am not sure if this is a good idea.
>
> Jackrabbit 2 had a single thread, which had the issue of one observation listener blocking all others.
>
> An obvious solution would be a thread pool (probably configurable number of threads). And on top of that, if the pool is full and no thread is free anymore, one could simply kill blocked handlers after a certain timeout (and optionally blacklist them).
>
> 5) filtering in central thread?
>
> There was some discussion about the current filtering evaluation [0]. Each listener has its own ChangeProcessor which in turn has its own ChangeDispatcher.Listener. This means filtering is not in a single, central thread, but happens in each listener's thread (they each have their own ChangeSet queue, which indeed might be different because the previous root NodeState to compare to might be different for each).
>
> I thought this was not optimal but it seems I was wrong - splitting that up and generating events in a central thread and then passing them in a queue to the listeners actually turned out to scale worse (in a quick test with many parallel listeners). See my patch at [1].
>
> I guess it is that reading asynchronously from the immutable NodeStates is more efficient than multiple blocking queues. Which speaks good for the underlying oak implementation :)
>
> [0] http://markmail.org/thread/533orsfr44wllvrx
> [1] https://github.com/alexkli/jackrabbit-oak/commit/aee631ded4996194a3ad0dec1fc7a9917f7123b8
>
> 6) long-running session just for observation
>
> One problem with JCR observation is that you need one session open all the time "just for the observation". This is done to be able to run observation under the permissions of a certain user. But nobody in practice uses the session for anything else than reading data upon events (which mostly is done to filter only, see 2); when you need to write things in an event, best practice is to create a new session on demand an close it again. With Oak's refresh() policy this is even more important.
>
> Maybe a new API from 2) could skip the requirement of an open session. Registration would be based upon a session to handle the credentials easily, but that session could be closed without killing the listener. You would get back a special listener object that would need to be kept alive and referenced to be able to clean up listeners automatically on a finalize() if they forget to unregister.
>
> Or you keep the session, but mark it in a special way so that Oak can internally save resources. Or maybe the session in oak is already so light-weight that this is actually no problem at all. I am just mentioning this because of the many warnings I get from oak in the logs about sessions that are open very long (and don't call refresh() IIRC).
>
> 7) move code to oak-jcr (minor)
>
> Quite a bit of the code in oak.plugins.observation is currently JCR eventing specific (EventGenerator for example), afaics this is better suited in oak-jcr. Although with 2) it might change a bit anyway.
>
> Cheers,
> Alex
>
Re: Observation
Posted by Alexander Klimetschek <ak...@adobe.com>.
On 07.11.2013, at 10:44, Jukka Zitting <ju...@gmail.com> wrote:
> Right, but the local-only observation pattern is structurally flawed,
> as it can't guarantee that the events actually do get processed (for
> example if the server dies during event delivery).
Observation can't guarantee everything, at some point the application needs to handle that, e.g. manual refresh option or users simply repeating the task. And I thought we agreed on removing those nasty cluster events as much as possible :) IMHO this is a place to trade off 100% availability for scalability.
> Commit hooks don't have that problem.
Switching code to commit hooks and making them very Oak dependent is quite an effort, if that's what you mean. Or do you suggest to implement some similar simple listener mechanism based on commit hooks (but not forcing applications to write them themselves, coding against the oak internal API within the listener)?
As mentioned, ideally that new API I propose can be hooked into the existing JCR API with just an additional listener subtype interface (say in jackrabbit-api) and could be implemented transparently for Jackrabbit 2 (well, with some utility). Since we say the long-running session is no issue at all, we don't need a completely separate API to manage that.
> AFAICT there's no use case for which the local only -listener pattern
> would be the best solution, so while we do need it for backwards
> compatibility, it IMHO should not be the default for any new
> functionality.
>From an application perspective, I see it completely opposite :) Cluster events mostly only apple to core infrastructure (e.g. the code deployment in Sling), but not the high-level content processing for which observation is great. Those are local by default and are usually the majority, at least in our CMS environment.
Cheers,
Alex
Re: Observation
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Thu, Nov 7, 2013 at 1:12 PM, Tobias Bocanegra <tr...@apache.org> wrote:
> On Thu, Nov 7, 2013 at 5:46 AM, Jukka Zitting <ju...@gmail.com> wrote:
>> On Wed, Nov 6, 2013 at 8:49 PM, Alexander Klimetschek
>> <ak...@adobe.com> wrote:
>>> 1) listeners should not get external events by default
>>
>> I'd rather not make such a default. A component designed to process
>> local changes should in most cases be implemented as a commit hook,
>> not an observation listener.
>
> I disagree. the commit hook is executed before the commit - which
> still can fail. Also, the commit hook is highly inconvenient to use.
> what you really want are nicely aggregated observation events :-)
Right, but the local-only observation pattern is structurally flawed,
as it can't guarantee that the events actually do get processed (for
example if the server dies during event delivery). Commit hooks don't
have that problem.
The case where a commit hook would not be applicable is when the
relevant functionality triggers some external changes, for example
sending an email notification. In such cases the local-only
observation pattern is used as a basic cluster synchronization
mechanism for guaranteeing that the event is only processed once
across the cluster. There are other, more reliable ways of doing that.
For example the checkpoint mechanism used by the asynchronous indexer
can guarantee that each change is processed once and only once across
a cluster, even if there are multiple concurrent listeners or if all
of them become unavailable for a while.
AFAICT there's no use case for which the local only -listener pattern
would be the best solution, so while we do need it for backwards
compatibility, it IMHO should not be the default for any new
functionality.
BR,
Jukka Zitting
Re: Observation
Posted by Tobias Bocanegra <tr...@apache.org>.
On Thu, Nov 7, 2013 at 5:46 AM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Wed, Nov 6, 2013 at 8:49 PM, Alexander Klimetschek
> <ak...@adobe.com> wrote:
>> 1) listeners should not get external events by default
>
> I'd rather not make such a default. A component designed to process
> local changes should in most cases be implemented as a commit hook,
> not an observation listener.
I disagree. the commit hook is executed before the commit - which
still can fail. Also, the commit hook is highly inconvenient to use.
what you really want are nicely aggregated observation events :-)
regards, toby
> And for backwards compatibility with
> existing JCR listeners we in any case need to keep the external events
> included by default.
>
>> 4) listener threads
>>
>> Currently in oak each event listener gets its own thread. In our app CQ
>> we have about 150 listeners, so you end up introducing 150 threads.
>> I am not sure if this is a good idea.
>
> I would say the problem here is the number of listeners, not the
> number of threads.
>
>> An obvious solution would be a thread pool (probably configurable number of
>> threads). And on top of that, if the pool is full and no thread is free anymore,
>> one could simply kill blocked handlers after a certain timeout (and
>> optionally blacklist them).
>
> I don't see a major benefit over simply having a separate thread for
> each listener, just a extra complexity, especially since killing
> blocked threads is nasty business in Java.
>
>> 5) filtering in central thread?
>> [...]
>> I guess it is that reading asynchronously from the immutable NodeStates
>> is more efficient than multiple blocking queues. Which speaks good for
>> the underlying oak implementation :)
>
> Right, that's what it was designed for. :-) See my post
> (http://markmail.org/message/ckmphhdfehbb3qkw) in the related thread
> for an explanation; the incremental cost of doing a repeated content
> diff is much smaller than that of the first diff, so especially on a
> system with multiple cores it makes more sense to do a bit of repeated
> work in parallel instead of relying on a central thread for
> pre-processing. The added benefit is that there is no risk of an event
> queue to grow without bounds because of a blocked listener.
>
>> 6) long-running session just for observation
>> [...]
>> Or you keep the session, but mark it in a special way so that Oak
>> can internally save resources. Or maybe the session in oak is already
>> so light-weight that this is actually no problem at all.
>
> The latter. Unlike with Jackrabbit 2.x, adding a new session does not
> slow down all the existing ones. The warning for long-lived sessions
> is there just to remind people to use refresh() in such cases (though
> with the auto-refresh mode the warning is kind of moot).
>
>> 7) move code to oak-jcr (minor)
>
> Agreed.
>
> BR,
>
> Jukka Zitting
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
On 7.11.13 2:46 , Jukka Zitting wrote:
>> An obvious solution would be a thread pool (probably configurable number of
>> >threads). And on top of that, if the pool is full and no thread is free anymore,
>> >one could simply kill blocked handlers after a certain timeout (and
>> >optionally blacklist them).
> I don't see a major benefit over simply having a separate thread for
> each listener, just a extra complexity, especially since killing
> blocked threads is nasty business in Java.
>
See my reply to Alex. I wouldn't go so far as to kill blocked listeners.
But having a deployment controlled thread pool - especially if we could
integrate this with the whiteboard - would definitely make this easier
to control.
Michael
Re: Observation
Posted by Michael Dürig <md...@apache.org>.
On 7.11.13 2:46 , Jukka Zitting wrote:
>> 4) listener threads
>> >
>> >Currently in oak each event listener gets its own thread. In our app CQ
>> >we have about 150 listeners, so you end up introducing 150 threads.
>> >I am not sure if this is a good idea.
> I would say the problem here is the number of listeners, not the
> number of threads.
>
Which better filtering support might help to reduce. E.g. listen to
multiple paths with a single listener.
Michael
Re: Observation
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, Nov 6, 2013 at 8:49 PM, Alexander Klimetschek
<ak...@adobe.com> wrote:
> 1) listeners should not get external events by default
I'd rather not make such a default. A component designed to process
local changes should in most cases be implemented as a commit hook,
not an observation listener. And for backwards compatibility with
existing JCR listeners we in any case need to keep the external events
included by default.
> 4) listener threads
>
> Currently in oak each event listener gets its own thread. In our app CQ
> we have about 150 listeners, so you end up introducing 150 threads.
> I am not sure if this is a good idea.
I would say the problem here is the number of listeners, not the
number of threads.
> An obvious solution would be a thread pool (probably configurable number of
> threads). And on top of that, if the pool is full and no thread is free anymore,
> one could simply kill blocked handlers after a certain timeout (and
> optionally blacklist them).
I don't see a major benefit over simply having a separate thread for
each listener, just a extra complexity, especially since killing
blocked threads is nasty business in Java.
> 5) filtering in central thread?
> [...]
> I guess it is that reading asynchronously from the immutable NodeStates
> is more efficient than multiple blocking queues. Which speaks good for
> the underlying oak implementation :)
Right, that's what it was designed for. :-) See my post
(http://markmail.org/message/ckmphhdfehbb3qkw) in the related thread
for an explanation; the incremental cost of doing a repeated content
diff is much smaller than that of the first diff, so especially on a
system with multiple cores it makes more sense to do a bit of repeated
work in parallel instead of relying on a central thread for
pre-processing. The added benefit is that there is no risk of an event
queue to grow without bounds because of a blocked listener.
> 6) long-running session just for observation
> [...]
> Or you keep the session, but mark it in a special way so that Oak
> can internally save resources. Or maybe the session in oak is already
> so light-weight that this is actually no problem at all.
The latter. Unlike with Jackrabbit 2.x, adding a new session does not
slow down all the existing ones. The warning for long-lived sessions
is there just to remind people to use refresh() in such cases (though
with the auto-refresh mode the warning is kind of moot).
> 7) move code to oak-jcr (minor)
Agreed.
BR,
Jukka Zitting
Re: Observation
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
Sounds good to me! I was originally against adding a new API, but now that
I see all the problems with the JCR observation API, I think it a very
good idea.
One possible addition: as far as I know, for "remove" events, some
observation listeners would like to read the old data. I think Oak could
provide that data, at least in most cases. Would it make sense to add this
feature? Would it be hard to support it? (This would be just for Oak, not
Jackrabbit 2.x.)
Regards,
Thomas
On 11/7/13 2:49 AM, "Alexander Klimetschek" <ak...@adobe.com> wrote:
>Hi everyone,
>
>having looked at and discussed observation recently (partly on our Adobe
>internal list with some of our application use cases, partly with Toby
>f2f), here are some things that are IMHO worthwile to look at. I hope I
>can contribute a bit, but before I get pulled away again, I wanted at
>least summarize and persist() my ideas:
>
>1) listeners should not get external events by default
>
>OAK-1121 helps, but applications must say they exclude external events,
>while it should be the other way around (only local by default) - this
>can arguably not change because of the jcr observation contract. The
>current implementation can still be optimized a bit, see my comment in
>the issue.
>
>2) filtering happens too late within listeners because jcr observation
>API is too simple (OAK-1133, "observation listener plus")
>
>The filter logic would be passed declaratively and allow more options as
>noted in OAK-1133:
>
> addEventListener(listener, Filter.Path("/libs"),
>Filter.PropertyValue("jcr:content/sling:resourceType", "myValue"), ...)
>
>(just a rough sketch; the simplest solution could simply introduce a
>FilterEventListener interface that extends the existing EventListener and
>provides the filter in a getFilter() method; a simple instanceof check
>within oak-jcr could detect that; but a completely separate API might
>also make sense, see point 6 below)
>
>This could then be evaluated when the NodeStateDiff is calculated. What
>is good now already is that the base path (if specified in the jcr
>observation) is evaluated as soon as getting the diff roots. But many
>real use cases might want to have multiple paths (/apps OR /libs) or
>globbing (/content/site/*/jcr:content) - this could be easily and
>efficiently added here in one place (amongst those other filters). Having
>one or few base paths is in the oak case probably the best way to filter
>things and one should avoid listeners that register at root, and only
>filter on node types for example, which requires a lot more tree walking.
>
>In our app (CQ) we have 18 listeners that register for all events,
>because the JCR observation doesn't give them the right filter options,
>so they have to do it themselves. I expect them to be vastly improved
>with this new approach.
>
>Also, getting events for deleted nodes could be supported here, since
>there is full access to the old NodeState.
>
>The ideal solution could also be built backwards compatible: implement
>the same filtering with normal JCR API usage and automatically wrap the
>listener in another event listener that filters the "manual" way. This
>allows application code that works with both Jackrabbit 2 or Oak with
>minimal effort (would need some helper in jackrabbit-jcr-commons or so).
>Unless we want to backport this to Jackrabbit 2 as well.
>
>3) cluster events
>
>As agreed, external events can be the real scalability issue. It was also
>noted, that many cases do not need it at all (hence 1). But there are
>still cases that need it: in Sling for example, code deployment (bundles,
>JSP scripts, etc.) is based on the JCR repository, meaning when you put a
>bundle in a certain "install" folder in the JCR, all cluster instances
>need to pick it up. However, such deployment events are comparably rare
>(except on a developer machine :)), so the throughput doesn't have to be
>high. OTOH large-scale content changes will (and should) be handled
>locally, thus not requiring external propagation.
>
>Now the question is if 2) with a filter-as-detailed-and-early-as-possible
>approach solves this already (having the right external/non-external flag
>on all listeners, plus detailed filters for the external listeners), or
>if we could make use of the fairly rare nature of those events and try to
>filter on the _source_ instance already and propagate those few events in
>some other, efficient way to other cluster instances. This might improve
>performance as the ChangeDispatcher#externalChange() check (polling all
>100ms) could then be removed completely. But if the filtering on the
>NodeStateDiff is fast (because listeners are specific), then maybe it is
>ok.
>
>4) listener threads
>
>Currently in oak each event listener gets its own thread. In our app CQ
>we have about 150 listeners, so you end up introducing 150 threads. I am
>not sure if this is a good idea.
>
>Jackrabbit 2 had a single thread, which had the issue of one observation
>listener blocking all others.
>
>An obvious solution would be a thread pool (probably configurable number
>of threads). And on top of that, if the pool is full and no thread is
>free anymore, one could simply kill blocked handlers after a certain
>timeout (and optionally blacklist them).
>
>5) filtering in central thread?
>
>There was some discussion about the current filtering evaluation [0].
>Each listener has its own ChangeProcessor which in turn has its own
>ChangeDispatcher.Listener. This means filtering is not in a single,
>central thread, but happens in each listener's thread (they each have
>their own ChangeSet queue, which indeed might be different because the
>previous root NodeState to compare to might be different for each).
>
>I thought this was not optimal but it seems I was wrong - splitting that
>up and generating events in a central thread and then passing them in a
>queue to the listeners actually turned out to scale worse (in a quick
>test with many parallel listeners). See my patch at [1].
>
>I guess it is that reading asynchronously from the immutable NodeStates
>is more efficient than multiple blocking queues. Which speaks good for
>the underlying oak implementation :)
>
>[0] http://markmail.org/thread/533orsfr44wllvrx
>[1]
>https://github.com/alexkli/jackrabbit-oak/commit/aee631ded4996194a3ad0dec1
>fc7a9917f7123b8
>
>6) long-running session just for observation
>
>One problem with JCR observation is that you need one session open all
>the time "just for the observation". This is done to be able to run
>observation under the permissions of a certain user. But nobody in
>practice uses the session for anything else than reading data upon events
>(which mostly is done to filter only, see 2); when you need to write
>things in an event, best practice is to create a new session on demand an
>close it again. With Oak's refresh() policy this is even more important.
>
>Maybe a new API from 2) could skip the requirement of an open session.
>Registration would be based upon a session to handle the credentials
>easily, but that session could be closed without killing the listener.
>You would get back a special listener object that would need to be kept
>alive and referenced to be able to clean up listeners automatically on a
>finalize() if they forget to unregister.
>
>Or you keep the session, but mark it in a special way so that Oak can
>internally save resources. Or maybe the session in oak is already so
>light-weight that this is actually no problem at all. I am just
>mentioning this because of the many warnings I get from oak in the logs
>about sessions that are open very long (and don't call refresh() IIRC).
>
>7) move code to oak-jcr (minor)
>
>Quite a bit of the code in oak.plugins.observation is currently JCR
>eventing specific (EventGenerator for example), afaics this is better
>suited in oak-jcr. Although with 2) it might change a bit anyway.
>
>Cheers,
>Alex
>