[jr3] Repository microkernel

[Jukka Zitting <ju...@gmail.com>]

Hi,

Assuming we have a unified persistence layer (see the other thread),
it should be possible to implement a relatively simple repository
"microkernel" that implements just enough functionality for higher
level components like search, versioning, node types, observation,
etc. to be implemented on top of it.

Such a microkernel would support two main goals:

* modularity: It would be easy to use things like OSGi to replace or
plug in different higher level components.

* portability: It would be possible to port the microkernel to other
environments like PHP, Ruby or a plain old Apache module in C

Achieving this would probably require fairly extensive changes to our
current architecture.

Do you think this would be worth the effort, and (if yes) how should
we approach this task?

BR,

Jukka Zitting

Re: [jr3] Repository microkernel

[Thomas Müller <th...@day.com>]

Hi,

>> - No change merging. Instead, use MVCC (when reading) and 'node level
>> write locking'. As far as I know this is like most MVCC databases
>> work. Actually we could use 'property level write locking'.
>
> right, but that has the major drawback that we cannot write transient
> modifications directly into the backend. unless we limit long running
> transient-scoped changes with a timer. otherwise a transient
> modification could easily hold up other sessions. besides, how would
> you support node level write locks in a clustered environment?

> transient modification could easily hold up other sessions

Only if both want to write to the same node. I believe this is
acceptable within the micro-kernel.

> how would you support node level write locks in a clustered environment?
> I think there is no way around some change merging.

Yes, but it's not necessarily the task of the micro-kernel to merge
changes. I suggest that the micro-kernel throws an exception in this
case (or asks the "callback mechanism" what to do), and the cluster
synchronization mechanism (or whatever is responsible; maybe it's the
application, maybe Jackrabbit) merges the changes then. Or re-runs the
operation.

> I would try to design something that works along the lines of a SCM system. before
> you commit, do an update and merge external changes into your
> modification, then commit. I think for most of the conflict scenarios
> there are sensible resolutions that will work well in practice.

This doesn't work for some cases, for example for reservation systems.

I don't think we need to solve "long running transactions" (that run
for hours). Those should be split into smaller transactions.

Most transactions are very small. Some are quite big (large imports,
large deletes). We do need to support those without running out of
memory, but I don't think we need to support automatic merging of
concurrent writes (at least not within the micro-kernel).

Regards,
Thomas

Re: [jr3] Repository microkernel

[Alexander Klimetschek <ak...@day.com>]

On Fri, Feb 19, 2010 at 16:03, Marcel Reutegger
<ma...@gmx.net> wrote:
> I think there is no way around some change merging. I would try to
> design something that works along the lines of a SCM system. before
> you commit, do an update and merge external changes into your
> modification, then commit. I think for most of the conflict scenarios
> there are sensible resolutions that will work well in practice.

Maybe we should look if we can learn something from git, which is
fast. The article Thomas pointed at was already interesting.

Regards,
Alex

-- 
Alexander Klimetschek
alexander.klimetschek@day.com

Re: [jr3] Repository microkernel

[Marcel Reutegger <ma...@gmx.net>]

hi,

On Fri, Feb 19, 2010 at 11:12, Thomas Müller <th...@day.com> wrote:
> Hi,
>
> A agree with Marcel, the current Jackrabbit SPI it too high level.
>
>> it must be impossible to create inconsistent data using the micro-kernel API
>> - tree and referential integrity awareness
>
> +1. No more consistency check / fix. No more inconsistent index if the
> property-value-index is also part of the micro-kernel.
>
>> - long running transaction support
>
> I think 'large' transaction support is important, but it doesn't need
> to be very fast. I would avoid creating temporary files (persisting
> the cache). Instead, changes could be written to storage, but not
> committed.
>
> A few ideas (but I'm not convinced all of them are good ideas):
>
> - In memory, each node points to its (main) parent. If a node is in
> memory, then its parent is also.
>
> - Nodes are immutable (in memory, and on disk). Each change (or at
> least commit) will replace all parent nodes including the root node.
>
> - No change merging. Instead, use MVCC (when reading) and 'node level
> write locking'. As far as I know this is like most MVCC databases
> work. Actually we could use 'property level write locking'.

right, but that has the major drawback that we cannot write transient
modifications directly into the backend. unless we limit long running
transient-scoped changes with a timer. otherwise a transient
modification could easily hold up other sessions. besides, how would
you support node level write locks in a clustered environment?

I think there is no way around some change merging. I would try to
design something that works along the lines of a SCM system. before
you commit, do an update and merge external changes into your
modification, then commit. I think for most of the conflict scenarios
there are sensible resolutions that will work well in practice.

regards
 marcel

> - Support multiple persistence backends (database, file system,...).
>
> - Support two phase commit and distributed transaction at a very low
> level, so that it's very easy to distribute data to many storage
> backends.
>
> - Move operations are copy+delete internally (maybe reordering a node
> in the list of child nodes is also a move).
>
> - Subtrees that didn't change for a longer time are eventually
> persisted as one blob in the data store in a form that is compact and
> fast to read
>
> Regards,
> Thomas
>

Re: [jr3] Repository microkernel

[Thomas Müller <th...@day.com>]

Hi,

A agree with Marcel, the current Jackrabbit SPI it too high level.

> it must be impossible to create inconsistent data using the micro-kernel API
> - tree and referential integrity awareness

+1. No more consistency check / fix. No more inconsistent index if the
property-value-index is also part of the micro-kernel.

> - long running transaction support

I think 'large' transaction support is important, but it doesn't need
to be very fast. I would avoid creating temporary files (persisting
the cache). Instead, changes could be written to storage, but not
committed.

A few ideas (but I'm not convinced all of them are good ideas):

- In memory, each node points to its (main) parent. If a node is in
memory, then its parent is also.

- Nodes are immutable (in memory, and on disk). Each change (or at
least commit) will replace all parent nodes including the root node.

- No change merging. Instead, use MVCC (when reading) and 'node level
write locking'. As far as I know this is like most MVCC databases
work. Actually we could use 'property level write locking'.

- Support multiple persistence backends (database, file system,...).

- Support two phase commit and distributed transaction at a very low
level, so that it's very easy to distribute data to many storage
backends.

- Move operations are copy+delete internally (maybe reordering a node
in the list of child nodes is also a move).

- Subtrees that didn't change for a longer time are eventually
persisted as one blob in the data store in a form that is compact and
fast to read

Regards,
Thomas

Re: [jr3] Repository microkernel

[Marcel Reutegger <ma...@gmx.net>]

hi,

I like the idea of a micro kernel. However, I don't think that the SPI
is a good starting point. it contains the full feature set of JCR. I
guess the current interface that best matches the micro-kernel idea is
the PersistenceManager interface. It strictly deals with persistence,
which is something that the micro-kernel implementation should do as
well, IIUC.

the gaps that I see are:
- tree and referential integrity awareness, it must be impossible to
create inconsistent data using the micro-kernel API (currently a
misbehaving client of the PersistenceManager API can currupt data)
- may behave eventual consistent. what's the effect on the micro-kernel API?
- support read-only multi-versioning. how are versions identified?
transaction id, timestamp?
- low-level copy/clone (are there other workspace operations that
would benefit from micro-kernel API support?)
- long running transaction support? I'm not sure about this one. the
use case here is writing transient modification directly to the
micro-kernel. I think Jukka thought about this already when he
prototyped his Next Gen Persistence a while ago. I think this would
come in very handy when we want to support large transactions. see
other thread. from an implementation perspective, we'd need to use
optimistic concurrency control (like we do currently) and merge
overlapping changes whenever possible.

regards
 marcel

On Thu, Feb 18, 2010 at 13:13, Alexander Klimetschek <ak...@day.com> wrote:
>> On Wed, Feb 17, 2010 at 4:48 PM, Jukka Zitting <ju...@gmail.com> wrote:
>>> Assuming we have a unified persistence layer (see the other thread),
>>> it should be possible to implement a relatively simple repository
>>> "microkernel" that implements just enough functionality for higher
>>> level components like search, versioning, node types, observation,
>>> etc. to be implemented on top of it.
>
> +1
>
> On Thu, Feb 18, 2010 at 11:50, Stefan Guggisberg
> <st...@gmail.com> wrote:
>> absolutely. the SPI would IMO be an ideal starting point for such a
>> microkernel api. we might need to review/strip it though.
>
> +1
>
> I think we should start with a scalable persistence layer (as
> discussed in the other threads) and from there find the simplest API
> needed for it, and then merge it with the existing SPI. An important
> goal for that microkernel API would be that you can implement it
> within one day (functionally, not necessarily fast), as compared with
> the full JCR API, that requires many things like node types,
> observation, shareable nodes, versioning etc.
>
> Regards,
> Alex
>
> --
> Alexander Klimetschek
> alexander.klimetschek@day.com
>

Re: [jr3] Repository microkernel

[Alexander Klimetschek <ak...@day.com>]

> On Wed, Feb 17, 2010 at 4:48 PM, Jukka Zitting <ju...@gmail.com> wrote:
>> Assuming we have a unified persistence layer (see the other thread),
>> it should be possible to implement a relatively simple repository
>> "microkernel" that implements just enough functionality for higher
>> level components like search, versioning, node types, observation,
>> etc. to be implemented on top of it.

+1

On Thu, Feb 18, 2010 at 11:50, Stefan Guggisberg
<st...@gmail.com> wrote:
> absolutely. the SPI would IMO be an ideal starting point for such a
> microkernel api. we might need to review/strip it though.

+1

I think we should start with a scalable persistence layer (as
discussed in the other threads) and from there find the simplest API
needed for it, and then merge it with the existing SPI. An important
goal for that microkernel API would be that you can implement it
within one day (functionally, not necessarily fast), as compared with
the full JCR API, that requires many things like node types,
observation, shareable nodes, versioning etc.

Regards,
Alex

-- 
Alexander Klimetschek
alexander.klimetschek@day.com

Re: [jr3] Repository microkernel

[Stefan Guggisberg <st...@gmail.com>]

On Wed, Feb 17, 2010 at 4:48 PM, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> Assuming we have a unified persistence layer (see the other thread),
> it should be possible to implement a relatively simple repository
> "microkernel" that implements just enough functionality for higher
> level components like search, versioning, node types, observation,
> etc. to be implemented on top of it.
>
> Such a microkernel would support two main goals:
>
> * modularity: It would be easy to use things like OSGi to replace or
> plug in different higher level components.
>
> * portability: It would be possible to port the microkernel to other
> environments like PHP, Ruby or a plain old Apache module in C

+1

>
> Achieving this would probably require fairly extensive changes to our
> current architecture.
>
> Do you think this would be worth the effort, and (if yes) how should
> we approach this task?

absolutely. the SPI would IMO be an ideal starting point for such a
microkernel api. we might need to review/strip it though.

cheers
stefan

>
> BR,
>
> Jukka Zitting
>