Re: Pull request: multiplexing NodeStore

[Michael Dürig <md...@apache.org>]

Hi Robert,

As this is a rather big pull request I think it would be good to shed 
some light on the background:

* What are the use cases / driver you have in mind for this feature? I 
would like to document these early on to avoid mission creep.

* Where are the limitation and how are we going to deal with them? 
Specifically: how do you deal with atomicity when multiple node stores 
are mounted? Similarly, how do you guarantee global constraints (like 
uniqueness).

* How is (un)mounting handled? How do you ensure the node store satisfy 
all higher level constraints?

Michael



On 30.9.16 3:45 , Robert Munteanu wrote:
> Hi,
>
> I've submitted a pull request for a multiplexing NodeStore based on the
> work done my Tomek and myself.
>
>   https://github.com/apache/jackrabbit-oak/pull/55
>   https://issues.apache.org/jira/browse/OAK-4871
>
> While it's not 100% done ( see Jira/PR for more details ) I believe
> that since it's isolated from the rest of the code base it would be
> beneficial for further maintenance and a high-level design check to
> submit it early rather than when it's 'done'.
>
> Comments towards what is still needed to actually incorporate it in
> oak-core would be very appreciated.
>
> Thanks,
>
> Robert
>

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

On Thu, 2016-10-27 at 09:47 +0200, Michael D�rig wrote:
> 
> Hi Robert,
> 
> On 26.10.16 11:28 , Robert Munteanu wrote:
> > On Wed, 2016-10-05 at 12:13 +0300, Robert Munteanu wrote:
> >>
> >> Created a sub-task for this
> >>
> >>��� OAK-4891 - Mount-time sanity checks for mounted NodeStore
> >> instances
> >>��� https://issues.apache.org/jira/browse/OAK-4891
> >
> > I've thought about this a little more. When mounting a NodeStore we
> are
> > working (unsurprisingly) at the NodeStore level, so we don't have
> any
> > concept of node types, versions, etc. So I can't use any of the
> logic
> > from oak-jcr, like the VersionManagerImpl.
> 
> This kind of logic is implemented in the various commit hooks. See
> e.g.�
> {{org.apache.jackrabbit.oak.plugins.version.VersionHook}}. The
> sanity�
> checks should reuse these hooks.

Thanks for the pointer to the hooks. While the VersionHook seems to do
a lot more than I need, in this particular scenario my needs are
satisfied by 
o.a.j.o.plugins.version.ReadOnlyVersionManger#isVersionable . I'll
look for classes implementing the functionality that I need for the
other checks as well.

Robert

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

On Thu, 2016-10-27 at 12:53 +0000, Tomek Rekawek wrote:
> Hi Robert,
> 
> > On 27 Oct 2016, at 14:11, Robert Munteanu <ro...@apache.org>
> > wrote:
> > 
> > If we have referenceable nodes or versionable nodes in the
> > 'private'
> > repository then it will be inconsistent, as they point to data
> > maintained under the 'global' store. So we need to prevent such
> > 'corrupt' mounts from being used.
> 
> Versionable - agreed, we can\u2019t have mix:versionable in the \u201cprivate\u201d
> repository.
> 
> Referenceable - not sure here. All nt:files are refernceable by
> default. Maybe it would be enough to forbid cross-mount references
> and rebuild the UUID index if the multiplexing setup changes?

That's one approach. Another is to use the oak:resource node type
instead of nt:resource. But the point is that we need these checks in
place.

I think Chetan was looking into the oak:resource independent of the
multiplexing initiative so we might inherit it for free.

But AFAICT this is the maximal check list:

- no versionable nodes
- no referenceable nodes

We can add more if needed in the future anyway.

Robert

Re: Pull request: multiplexing NodeStore

[Tomek Rekawek <re...@adobe.com>]

Hi Robert,

> On 27 Oct 2016, at 14:11, Robert Munteanu <ro...@apache.org> wrote:
> 
> If we have referenceable nodes or versionable nodes in the 'private'
> repository then it will be inconsistent, as they point to data
> maintained under the 'global' store. So we need to prevent such
> 'corrupt' mounts from being used.

Versionable - agreed, we can’t have mix:versionable in the “private” repository.

Referenceable - not sure here. All nt:files are refernceable by default. Maybe it would be enough to forbid cross-mount references and rebuild the UUID index if the multiplexing setup changes?

Regards,
Tomek

-- 
Tomek Rękawek | Adobe Research | www.adobe.com
rekawek@adobe.com

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

On Thu, 2016-10-27 at 10:58 +0000, Tomek Rekawek wrote:
> Hello,
> 
> let me describe the use cases we have in mind for the multiplexing
> node store in a more detailed way. I hope it\u2019ll allow to plan a few
> (well, two) milestones for the implementation.

Thanks for providing the extra context, Tomek.

(snip)

> Since the \u201cprivate\u201d and \u201cglobal\u201d repositories are really extracted
> parts of the same \u201cinitial\u201d repository, we don\u2019t need to worry about
> the consistency - as at the node store level \u201cprivate\u201d + \u201cglobal\u201d
> repository === \u201cinitial\u201d repository, which, as we know, is
> consistent.

If we have referenceable nodes or versionable nodes in the 'private'
repository then it will be inconsistent, as they point to data
maintained under the 'global' store. So we need to prevent such
'corrupt' mounts from being used.


> This scenario (a) is possible to have with the current version of the
> multiplexing node store patch. That\u2019s why I think it\u2019d be a good idea
> to document it (putting an emphasis on fact that splitting an
> existing repository into parts using oak-upgrade is right now the
> *only* way to get a consistent multiplexing node store repository)
> and merge. The patch is already quite big and I think that postponing
> it will make it hard to track the progress here.

I second that, the patch set is quite large, and we will need to build
on it anyway. I would suggest clarifying as soon as possible:

1. If the current design is something that can be incorporated in Oak (
I would hope so :-) ) .
2. What are the implementation changes ( if any ) that need to be done
for an initial contribution. Note I added checks for read-only mounts
as discussed after my initial email.

This will definitely see more fixes and tweaks in the following weeks,
but to get those in we need feedback from consuming parties, which is
much easier to get with an 'official' Oak SNAPSHOT build.

Thanks,

Robert

Re: Pull request: multiplexing NodeStore

[Tomek Rekawek <re...@adobe.com>]

Hello,

let me describe the use cases we have in mind for the multiplexing node store in a more detailed way. I hope it’ll allow to plan a few (well, two) milestones for the implementation.

Let’s assume a Sling installation, in which /apps and /libs nodes contains the immutable application code (component, JSPs, etc.) and all the other paths (/content, /var) contain the mutable application state. Our goal here is to extract the application code into a separate “private” repository, while keeping the instance state in the main “global” repo.

This gives us two new possibilities:

(a) the “private” repository is a low-latency one, so if the “global” repository is shared, distributed and therefore slower, we’ll have fast access to the application code that doesn’t need to be shared,
(b) the application can be upgraded to the newer version by simply switching the “private” repository.

In order to achieve (a) we need a way to split a normal (“initial”) Sling repository into private and global. It can be done using oak-upgrade:

1) we start a normal Sling instance. The multiplexing node store is not configured yet.
2) once the Sling starts, we stop it and use oak-upgrade to:
-copy /apps and /libs to the “private” repository,
-copy all the other paths to the “global” repository.
3) we start the Sling again, this time with multiplexing node store configured to support “private” and “global” repository.

Since the “private” and “global” repositories are really extracted parts of the same “initial” repository, we don’t need to worry about the consistency - as at the node store level “private” + “global” repository === “initial” repository, which, as we know, is consistent.

This scenario (a) is possible to have with the current version of the multiplexing node store patch. That’s why I think it’d be a good idea to document it (putting an emphasis on fact that splitting an existing repository into parts using oak-upgrade is right now the *only* way to get a consistent multiplexing node store repository) and merge. The patch is already quite big and I think that postponing it will make it hard to track the progress here.

In order to implement scenario (b) we’ll have to make sure that already existing “global” repository combined with the new “private” repository (created on an alien instance) is still consistent. This is the place where we have to perform all the sanity checks. I’d like, however, to make this the next phase of multiplexing node store implementation, rather than making this a part of the current patch (which is limited as in above, but correct, considering its constraints).

WDYT?

Regards,
Tomek

-- 
Tomek Rękawek | Adobe Research | www.adobe.com
rekawek@adobe.com

> On 27 Oct 2016, at 09:47, Michael Dürig <md...@apache.org> wrote:
> 
> 
> Hi Robert,
> 
> On 26.10.16 11:28 , Robert Munteanu wrote:
>> On Wed, 2016-10-05 at 12:13 +0300, Robert Munteanu wrote:
>>> 
>>> Created a sub-task for this
>>> 
>>>   OAK-4891 - Mount-time sanity checks for mounted NodeStore
>>> instances
>>>   https://issues.apache.org/jira/browse/OAK-4891
>> 
>> I've thought about this a little more. When mounting a NodeStore we are
>> working (unsurprisingly) at the NodeStore level, so we don't have any
>> concept of node types, versions, etc. So I can't use any of the logic
>> from oak-jcr, like the VersionManagerImpl.
> 
> This kind of logic is implemented in the various commit hooks. See e.g. {{org.apache.jackrabbit.oak.plugins.version.VersionHook}}. The sanity checks should reuse these hooks.
> 
>> 
>> The only way that I see right now is to use the NodeStore API and
>> 'manually' perform the needed checks, e.g. look for nodes with a
>> property named 'jcr:mixinTypes' which has a 'mix:referenceable' value.
> 
> Please don't. We need to stay DRY here. Everything else will fall apart going forward.
> 
>> 
>> Is this approach a viable one? Also, is this done somewhere else in the
>> Oak codebase? I'd like to avoid duplication.
> 
> The intention of the commit hook was to make them pluggable. Have a look at the Jcr and Oak builder classes and how we actually inject those hooks to implement the JCR semantics. The sanity checks should follow a similar approach IMO.
> 
> Michael
> 
>> 
>> Thanks,
>> 
>> Robert
>> 

Re: Pull request: multiplexing NodeStore

[Michael Dürig <md...@apache.org>]

Hi Robert,

On 26.10.16 11:28 , Robert Munteanu wrote:
> On Wed, 2016-10-05 at 12:13 +0300, Robert Munteanu wrote:
>>
>> Created a sub-task for this
>>
>>    OAK-4891 - Mount-time sanity checks for mounted NodeStore
>> instances
>>    https://issues.apache.org/jira/browse/OAK-4891
>
> I've thought about this a little more. When mounting a NodeStore we are
> working (unsurprisingly) at the NodeStore level, so we don't have any
> concept of node types, versions, etc. So I can't use any of the logic
> from oak-jcr, like the VersionManagerImpl.

This kind of logic is implemented in the various commit hooks. See e.g. 
{{org.apache.jackrabbit.oak.plugins.version.VersionHook}}. The sanity 
checks should reuse these hooks.

>
> The only way that I see right now is to use the NodeStore API and
> 'manually' perform the needed checks, e.g. look for nodes with a
> property named 'jcr:mixinTypes' which has a 'mix:referenceable' value.

Please don't. We need to stay DRY here. Everything else will fall apart 
going forward.

>
> Is this approach a viable one? Also, is this done somewhere else in the
> Oak codebase? I'd like to avoid duplication.

The intention of the commit hook was to make them pluggable. Have a look 
at the Jcr and Oak builder classes and how we actually inject those 
hooks to implement the JCR semantics. The sanity checks should follow a 
similar approach IMO.

Michael

>
> Thanks,
>
> Robert
>

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

On Wed, 2016-10-05 at 12:13 +0300, Robert Munteanu wrote:
> 
> Created a sub-task for this
> 
> �� OAK-4891 - Mount-time sanity checks for mounted NodeStore
> instances
> �� https://issues.apache.org/jira/browse/OAK-4891

I've thought about this a little more. When mounting a NodeStore we are
working (unsurprisingly) at the NodeStore level, so we don't have any
concept of node types, versions, etc. So I can't use any of the logic
from oak-jcr, like the VersionManagerImpl.

The only way that I see right now is to use the NodeStore API and
'manually' perform the needed checks, e.g. look for nodes with a
property named 'jcr:mixinTypes' which has a 'mix:referenceable' value.

Is this approach a viable one? Also, is this done somewhere else in the
Oak codebase? I'd like to avoid duplication.

Thanks,

Robert

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

On 2016-10-04 10:48, Michael Dürig wrote:
> On 3.10.16 4:33 , Robert Munteanu wrote:
>>> * Where are the limitation and how are we going to deal with them?
>>> > Specifically: how do you deal with atomicity when multiple node
>>> > stores
>>> > are mounted? Similarly, how do you guarantee global constraints
>>> > (like
>>> > uniqueness).
>> We explicitly set write multiplexing as a non-goal. This works for now
>> as a side-effect - it was too easy to implement :-)
>> 
>> So we will have at most ( usually exactly ) one store which is write-
>> enabled, and all other stores will be read-only. In the scenarios
>> above, /libs and /apps will be read-only.
>> 
>> At a JCR level we will make sure that the mounted repositories, except
>> the global one, will have no 'troublesome' nodes: no versioning, no
>> referenceable nodes, etc.
> 
> So we need to put some safeguards into place ensuring those
> prerequisites actually hold. If this cannot be done we at least need
> to clearly document them. What I want to avoid here is suddenly
> finding ourselves in the position where we are forced to support the
> "full" use case. Let's add the uses cases and constraints identified
> in this thread to the issue (or dedicated issues) so we don't forget.

Created a sub-task for this

   OAK-4891 - Mount-time sanity checks for mounted NodeStore instances
   https://issues.apache.org/jira/browse/OAK-4891

>> 
>> 
>>> > * How is (un)mounting handled? How do you ensure the node store
>>> > satisfy
>>> > all higher level constraints?
>> Mounting is configured via OSGi and expected to not change at runtime.
>> Doing this dynamically is a non-goal.
>> 
>> As for the other high-level constraints, a read-only mount with 
>> limited
>> JCR capabilities in terms of versioning, locking, etc should not pose
>> any issues. Can you maybe add some specific scenarios that you think
>> can be problematic?
> 
> This boils down to the same concerns as above: how do we make
> sufficiently sure that only such node stores are mounted which do not
> make use of these troublesome features? What happens if this is
> violated? (How) can we fail fast? I could imagine cases where this
> goes unnoticed for a long time but then suddenly starts failing in way
> very strange and hard to diagnose ways. For example: if accidentally a
> node store is mounted that violates the uniqueness constraint (e.g.
> UUID index). Can we detect this at mount time? What happens if not?

I think we should both ensure that when a mount is exported all 
troublesome
node structures are cleaned. Also, we should detect and fail fast at 
mount
time. I prefer a slightly slower mount time to runtime errors :-)

Thanks,

Robert

Re: Pull request: multiplexing NodeStore

[Michael Dürig <md...@apache.org>]

On 3.10.16 4:33 , Robert Munteanu wrote:
>> * Where are the limitation and how are we going to deal with them?
>> > Specifically: how do you deal with atomicity when multiple node
>> > stores
>> > are mounted? Similarly, how do you guarantee global constraints
>> > (like
>> > uniqueness).
> We explicitly set write multiplexing as a non-goal. This works for now
> as a side-effect - it was too easy to implement :-)
>
> So we will have at most ( usually exactly ) one store which is write-
> enabled, and all other stores will be read-only. In the scenarios
> above, /libs and /apps will be read-only.
>
> At a JCR level we will make sure that the mounted repositories, except
> the global one, will have no 'troublesome' nodes: no versioning, no
> referenceable nodes, etc.

So we need to put some safeguards into place ensuring those 
prerequisites actually hold. If this cannot be done we at least need to 
clearly document them. What I want to avoid here is suddenly finding 
ourselves in the position where we are forced to support the "full" use 
case. Let's add the uses cases and constraints identified in this thread 
to the issue (or dedicated issues) so we don't forget.

>
>
>> > * How is (un)mounting handled? How do you ensure the node store
>> > satisfy
>> > all higher level constraints?
> Mounting is configured via OSGi and expected to not change at runtime.
> Doing this dynamically is a non-goal.
>
> As for the other high-level constraints, a read-only mount with limited
> JCR capabilities in terms of versioning, locking, etc should not pose
> any issues. Can you maybe add some specific scenarios that you think
> can be problematic?

This boils down to the same concerns as above: how do we make 
sufficiently sure that only such node stores are mounted which do not 
make use of these troublesome features? What happens if this is 
violated? (How) can we fail fast? I could imagine cases where this goes 
unnoticed for a long time but then suddenly starts failing in way very 
strange and hard to diagnose ways. For example: if accidentally a node 
store is mounted that violates the uniqueness constraint (e.g. UUID 
index). Can we detect this at mount time? What happens if not?

Michael

Re: Pull request: multiplexing NodeStore

[Robert Munteanu <ro...@apache.org>]

Hi Michael,

On Mon, 2016-10-03 at 09:15 +0200, Michael D�rig wrote:
> Hi Robert,
> 
> As this is a rather big pull request I think it would be good to
> shed�
> some light on the background:
> 
> * What are the use cases / driver you have in mind for this feature?
> I�
> would like to document these early on to avoid mission creep.

From my point of view there are two main scenarios:

1. Combining a global store with a local one

When using a clustered DocumentStore instance it is sometimes useful to
have parts of the repository private to the instance - not shared.

For example, in Sling land, this would make rolling deployments
possible by making /libs and /apps private.

2. Faster container-based deployment by only packaging part of the
repository in the container image

When building container images the size and startup time should be
closely controlled. By allowing NodeStore multiplexing we can package
only /libs and /apps into the image and allow the rest of the
repository to live outside the container image.

> 
> * Where are the limitation and how are we going to deal with them?�
> Specifically: how do you deal with atomicity when multiple node
> stores�
> are mounted? Similarly, how do you guarantee global constraints
> (like�
> uniqueness).

We explicitly set write multiplexing as a non-goal. This works for now
as a side-effect - it was too easy to implement :-)

So we will have at most ( usually exactly ) one store which is write-
enabled, and all other stores will be read-only. In the scenarios
above, /libs and /apps will be read-only.

At a JCR level we will make sure that the mounted repositories, except
the global one, will have no 'troublesome' nodes: no versioning, no
referenceable nodes, etc.


> * How is (un)mounting handled? How do you ensure the node store
> satisfy�
> all higher level constraints?

Mounting is configured via OSGi and expected to not change at runtime.
Doing this dynamically is a non-goal.

As for the other high-level constraints, a read-only mount with limited
JCR capabilities in terms of versioning, locking, etc should not pose
any issues. Can you maybe add some specific scenarios that you think
can be problematic?

Thanks,

Robert
> 
> Michael
> 
> 
> 
> On 30.9.16 3:45 , Robert Munteanu wrote:
> > Hi,
> > 
> > I've submitted a pull request for a multiplexing NodeStore based on
> > the
> > work done my Tomek and myself.
> > 
> > � https://github.com/apache/jackrabbit-oak/pull/55
> > � https://issues.apache.org/jira/browse/OAK-4871
> > 
> > While it's not 100% done ( see Jira/PR for more details ) I believe
> > that since it's isolated from the rest of the code base it would be
> > beneficial for further maintenance and a high-level design check to
> > submit it early rather than when it's 'done'.
> > 
> > Comments towards what is still needed to actually incorporate it in
> > oak-core would be very appreciated.
> > 
> > Thanks,
> > 
> > Robert
> >