You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Marius Petria <mp...@adobe.com> on 2015/06/25 17:17:35 UTC
[Observation] Question on local events reported as external
Hi,
I understand that under high load local events can be reported as external. My question is why does this happen also for tarmk, where there is a single instance active (receiving external events on a single instance seems odd)? Also, is there a way to disable this functionality, meaning on tarmk to always receive the local events with their associated data (specifically the userData)?
Regards,
Marius
Re: [Observation] Question on local events reported as external
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Fri, Jun 26, 2015 at 1:49 AM, Marius Petria <mp...@adobe.com> wrote:
> The observation queue limit is per EventListener, meaning that I have to optimize only my listener in order to make it less likely for the limit to be hit.
Yes
> At least in theory (and with the exception of collapsing a creation and a deletion) the events are never dropped, right? The line in [1] even if it skips the current NodeState it will include it in a later revision, right?
Yes
Chetan Mehrotra
Re: [Observation] Question on local events reported as external
Posted by Marius Petria <mp...@adobe.com>.
Thanks Chetan for the detailed explanation. I have a couple more questions :).
1. The observation queue limit is per EventListener, meaning that I have to optimize only my listener in order to make it less likely for the limit to be hit.
2. At least in theory (and with the exception of collapsing a creation and a deletion) the events are never dropped, right? The line in [1] even if it skips the current NodeState it will include it in a later revision, right?
Marius
[1] https://github.com/apache/jackrabbit-oak/blob/80985b51a18ef61015d263d2358f5c3713fea873/oak-core/src/main/java/org/apache/jackrabbit/oak/spi/commit/BackgroundObserver.java#L278
On 6/25/15, 9:49 PM, "Chetan Mehrotra" <ch...@gmail.com> wrote:
>On Thu, Jun 25, 2015 at 10:49 PM, Marius Petria <mp...@adobe.com> wrote:
>> AFAIU because the local events are changed to external events it means that they can also be dropped completly under load, is that true?
>
>Well they are not dropped. Observation in Oak works on basis of diff
>of NodeState. So lets say you have an Observation queue of size 3 with
>3 local events
>
>1. [ns1, ns2, ci1]
>2. [ns2, ns3, ci2]
>3. [ns3, ns4, ci3]
>
>Each tuple is [base root nodestate, nodeState post change, commit
>info]. Now if 4 change event [ns4, ns5, ci4] comes the
>BackgroundObserver has following options
>
>1. Block on put
>2. Have a indefinite length queue then just add to queue
>3. Pull out the last event and replace it with merged
>
>So current logic in Oak goes for #3.
>
>1. [ns1, ns2, ci1]
>2. [ns2, ns3, ci2]
>3. [ns3, ns5, null]
>
>Now when it merged/collapsed the content change then you cannot
>associate any specific commit info to it. In such cases
>
>1. You cannot determine which user has made that change.
>2. Some changes might not get visible. For example if a foo property
>is added in E3 and removed in E4. Then Merged content change would not
>provide any indication that any such change happened
>
>So such merged changes are shown as external i.e.
>JackrabbitEvent#isExternal returns true for them currently. Problem is
>currently its not possible to distinguish such collapsed events from
>truely external events. May be we make that distinction so that
>component which just rely on *some local change* to react can continue
>to work. Though there is no gurantee that they see *each* local
>change.
>
>Chetan Mehrotra
Re: [Observation] Question on local events reported as external
Posted by Chetan Mehrotra <ch...@gmail.com>.
On Thu, Jun 25, 2015 at 10:49 PM, Marius Petria <mp...@adobe.com> wrote:
> AFAIU because the local events are changed to external events it means that they can also be dropped completly under load, is that true?
Well they are not dropped. Observation in Oak works on basis of diff
of NodeState. So lets say you have an Observation queue of size 3 with
3 local events
1. [ns1, ns2, ci1]
2. [ns2, ns3, ci2]
3. [ns3, ns4, ci3]
Each tuple is [base root nodestate, nodeState post change, commit
info]. Now if 4 change event [ns4, ns5, ci4] comes the
BackgroundObserver has following options
1. Block on put
2. Have a indefinite length queue then just add to queue
3. Pull out the last event and replace it with merged
So current logic in Oak goes for #3.
1. [ns1, ns2, ci1]
2. [ns2, ns3, ci2]
3. [ns3, ns5, null]
Now when it merged/collapsed the content change then you cannot
associate any specific commit info to it. In such cases
1. You cannot determine which user has made that change.
2. Some changes might not get visible. For example if a foo property
is added in E3 and removed in E4. Then Merged content change would not
provide any indication that any such change happened
So such merged changes are shown as external i.e.
JackrabbitEvent#isExternal returns true for them currently. Problem is
currently its not possible to distinguish such collapsed events from
truely external events. May be we make that distinction so that
component which just rely on *some local change* to react can continue
to work. Though there is no gurantee that they see *each* local
change.
Chetan Mehrotra
Re: [Observation] Question on local events reported as external
Posted by Marius Petria <mp...@adobe.com>.
AFAIU because the local events are changed to external events it means that they can also be dropped completly under load, is that true?
On 6/25/15, 6:57 PM, "Chetan Mehrotra" <ch...@gmail.com> wrote:
>This is most likely due to rate at which event gets generated and thus
>causing Observation queue to fill up (1000 default size). If the
>backing JCR listener is slow in processing the queue would get filled
>and BackgroundObserver would start compacting/merging the diff thus
>converting local events to external.
>
>There are two solutions
>1. Throttle the commits - CommitRateLimiter
>2. Have a non limiting queue - Then you end up with OOM if gap in
>processing rate is large
>Chetan Mehrotra
>
>
>On Thu, Jun 25, 2015 at 8:47 PM, Marius Petria <mp...@adobe.com> wrote:
>> Hi,
>>
>> I understand that under high load local events can be reported as external. My question is why does this happen also for tarmk, where there is a single instance active (receiving external events on a single instance seems odd)? Also, is there a way to disable this functionality, meaning on tarmk to always receive the local events with their associated data (specifically the userData)?
>>
>> Regards,
>> Marius
>>
Re: [Observation] Question on local events reported as external
Posted by Chetan Mehrotra <ch...@gmail.com>.
This is most likely due to rate at which event gets generated and thus
causing Observation queue to fill up (1000 default size). If the
backing JCR listener is slow in processing the queue would get filled
and BackgroundObserver would start compacting/merging the diff thus
converting local events to external.
There are two solutions
1. Throttle the commits - CommitRateLimiter
2. Have a non limiting queue - Then you end up with OOM if gap in
processing rate is large
Chetan Mehrotra
On Thu, Jun 25, 2015 at 8:47 PM, Marius Petria <mp...@adobe.com> wrote:
> Hi,
>
> I understand that under high load local events can be reported as external. My question is why does this happen also for tarmk, where there is a single instance active (receiving external events on a single instance seems odd)? Also, is there a way to disable this functionality, meaning on tarmk to always receive the local events with their associated data (specifically the userData)?
>
> Regards,
> Marius
>