Re: Effect of KernelRootBuilder update

[Michael Dürig <mi...@gmail.com>]

On 16.11.12 13:00, Jukka Zitting wrote:
> Hi,
>
> On Fri, Nov 16, 2012 at 2:48 PM, Michael Dürig <md...@apache.org> wrote:
>> I was having a look at KernelRootBuilder.update() which saves pending
>> changes down to a branch in the Microkernel after a certain threshold in the
>> number of changes. I was wondering if this has any effect at all. The branch
>> is never merged back. Or is that branch "only" used as a persistent cache?
>> Since there is similar logic in RootImpl already, this might also be work in
>> progress. If so, I suggest to disable it for the time being.
>
> Yes, we can disable it.

I was looking into this again in order to use it for replacing the purge 
logic we have in RootImpl. The nice part about this would be that higher 
layers wouldn't need to care about node builders growing "too large" 
since they would be backed by a branch in the Microkernel. However, I 
don't see a way to integrate this with NodeStoreBranch without breaking 
encapsulation.

>
> It's mostly an incomplete experiment at seeing whether we could get
> rid of the NodeStoreBranch feature entirely by pushing the existing
> RootImpl purge logic down below the NodeBuilder interface. Another
> rationale was to see how far we could go without having to rely on
> branches, as the MongoMK didn't yet support that feature.

Since we now have branches in the Microkernel I suggest, we remove this 
feature entirely. That is, remove the KernelNodeBuilder and 
KernelRootBuilder classes. Unless we want to further pursue getting rid 
of NodeStoreBranch, that is.

Michael

>
> I intentionally set the update limit high enough so that the
> functionality is only triggered in custom test cases and in normal use
> the much lower purge limit in RootImpl takes care of branch handling.
>
> BR,
>
> Jukka Zitting
>

Re: Effect of KernelRootBuilder update

[Michael Dürig <md...@apache.org>]

On 6.2.13 16:55, Michael Dürig wrote:

> Since we now have branches in the Microkernel I suggest, we remove this
> feature entirely. That is, remove the KernelNodeBuilder and
> KernelRootBuilder classes. Unless we want to further pursue getting rid
> of NodeStoreBranch, that is.
>

I created OAK-616 to track this.

Michael

Re: Effect of KernelRootBuilder update

[Michael Dürig <md...@apache.org>]

On 27.2.13 12:15, Jukka Zitting wrote:
> Hi,
>
> On Wed, Feb 6, 2013 at 6:55 PM, Michael Dürig <mi...@gmail.com> wrote:
>> Since we now have branches in the Microkernel I suggest, we remove this
>> feature entirely. That is, remove the KernelNodeBuilder and
>> KernelRootBuilder classes. Unless we want to further pursue getting rid of
>> NodeStoreBranch, that is.
>
> Actually we perhaps should consider this. A problem that I just
> encountered (see my post on large commits) is the way commit hooks
> operate directly on the NodeBuilder level. A large commit like a big
> import operation will trigger a large index update, which can easily
> lead to an OutOfMemoryError with the MemoryNodeBuilder.
>
> If we pushed the purge logic down below the NodeBuilder interface we
> could address that problem. As an additional benefit the underlying
> storage engine could better decide when and how pending changes should
> be persisted. For example the SegmentMK could decide to only persist
> those parts of the transient tree that haven't changed in a while,
> leaving the active parts still in memory.

Makes sense. I created https://issues.apache.org/jira/browse/OAK-659 to 
track this.

Michael

>
> BR,
>
> Jukka Zitting
>

Re: Effect of KernelRootBuilder update

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Feb 6, 2013 at 6:55 PM, Michael Dürig <mi...@gmail.com> wrote:
> Since we now have branches in the Microkernel I suggest, we remove this
> feature entirely. That is, remove the KernelNodeBuilder and
> KernelRootBuilder classes. Unless we want to further pursue getting rid of
> NodeStoreBranch, that is.

Actually we perhaps should consider this. A problem that I just
encountered (see my post on large commits) is the way commit hooks
operate directly on the NodeBuilder level. A large commit like a big
import operation will trigger a large index update, which can easily
lead to an OutOfMemoryError with the MemoryNodeBuilder.

If we pushed the purge logic down below the NodeBuilder interface we
could address that problem. As an additional benefit the underlying
storage engine could better decide when and how pending changes should
be persisted. For example the SegmentMK could decide to only persist
those parts of the transient tree that haven't changed in a while,
leaving the active parts still in memory.

BR,

Jukka Zitting