[jr3] Bundle format

[Thomas Müller <th...@day.com>]

I would like to define a new storage format for nodes and properties.
A few ideas:

== Name and Namespace Index ==

Currently each new property and node name is stored in the name index.
Each namespace is stored in the namespace index. Those indexes are
used to compress the data. There are several (smaller) problems with
this:

- The indexes are stored in properties file (non-transactional).
- In the past, there were a few problems when migrating data (copying
workspaces).
- Jackrabbit indexes *each* name and namespace. This can run out of
memory if there are many names (dynamically created names).
- This is a problem for clustering (specially when using the
eventually consistent model).

I would like to keep a name index mechanism for commonly used names
and namespaces, but would also support a non-indexed names / namespace
format. I think we should start with a fixed list. We could add a
mechanism to create new index entries later on.

== Node Id ==

Currently Jackrabbit uses UUIDs to identify nodes. Even nodes that are
not referenceable have UUIDs. This allows to create nodes
concurrently, which is good. It is not optimal for storage however
(index cache efficiency is very bad because the numbers are random;
size overhead). Also, it's quite in-flexible (its hard to refer to
external nodes).

For node id storage, I suggest to support multiple data types: UUID
(which is basically a fixed length or a string), long, and string. The
Jackrabbit implementation may not need to support all formats (at
least first), but the (bundle) storage format should.

== List of Parent Node Ids ==

I would store that as a (hidden) multi-value property.

== Commonly Used String ==

If we want to store node types as regular properties, we should avoid
storing the node type strings. Instead, we should store the node type
index only. This is similar to the name index and namespace index. I
suggest the storage format supports a set of indexed values (initially
a fixed list).

Regards,
Thomas

Re: [jr3] Bundle format

[Thomas Müller <th...@day.com>]

Hi,

I made a prototype for the Bundle and Value classes:
http://h2database.com/p.html#2f2ad854cbadd3b3c3676f6e02dc8058

A few remarks:

The Bundle class uses variable size integers / longs heavily. This
saves quite a lot of space. This is similar to SQLite, Protocol
Buffers, and H2. Some code is from
http://code.google.com/p/h2database/source/browse/trunk/h2/src/main/org/h2/store/Data.java

String are written almost like UTF-8 (this implementation is a bit
faster). Doubles are stored in IEEE 754 format, but bytes reversed,
and variable size. Short strings (0-15 characters) use one byte more
than the length. Date, decimal, references and so on are stored as
strings. Small binaries are stored in-place, for larger ones only the
reference. For "indexed values", only the index is stored (one or two
bytes).

Value class: There is a hardcoded list of "indexed values". The Value
class that supports multi-valued "values" ("array-of-values value").
There is a value cache for frequently used values that are not indexed
(to save memory - this idea is also from H2). Node names and property
names are also "Values" (actually two values for each name: the
namespace, plus the local name). The value class has 'deep' hashCode
and equals implementations, and is comparable.

A node could be stored as one "Value" instance, in the form of a
nested "array-of-values value". I will propose a node serialization
later on, but it's simply an array of properties (the child node list
is also a multi-valued property; everything is a property). Actually,
you could even store a tree of nodes as one "Value".

> We might also consider not storing each bundle in a separate record,
> instead making a single record of all the bundles included in a
> ChangeLog instance. Such super-bundles could perhaps be persisted in
> the data store.

Yes, that should be possible. Currently we keep the change log in
memory until the user calls "save". This is a problem for large
transactions (out of memory). I suggest to change this, so the system
(possibly also the user) can do some "temporary saves" that are not
committed yet. When using a database backend, this wouldn't keep a
database transaction open; instead it would store the changelog to a
new place ("new version") and then on Session.save() it would copy the
data over to the "main storage". More about that later on.

Regards,
Thomas

Re: [jr3] Bundle format

[Marcel Reutegger <ma...@gmx.net>]

On Tue, Feb 23, 2010 at 15:03, Jukka Zitting <ju...@gmail.com> wrote:
> Hi,
>
> On Sat, Feb 20, 2010 at 1:16 PM, Thomas Müller <th...@day.com> wrote:
>> I would like to define a new storage format for nodes and properties.
>
> More generally, I'd like to raise the bundle abstraction to a higher
> level. As a first step I'd make the PersistenceManager interface aware
> of bundles (i.e. load and store bundles instead of individual node and
> property states), and later perhaps replace the item state concept
> with bundle states to reduce the number of objects we need to keep
> track of.

+1 I like that idea.

regards
 marcel

> We might also consider not storing each bundle in a separate record,
> instead making a single record of all the bundles included in a
> ChangeLog instance. Such super-bundles could perhaps be persisted in
> the data store.
>
>> == Name and Namespace Index ==
>> [...]
>> == Commonly Used String ==
>
> +1
>
> BR,
>
> Jukka Zitting
>

Re: [jr3] Bundle format

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Sat, Feb 20, 2010 at 1:16 PM, Thomas Müller <th...@day.com> wrote:
> I would like to define a new storage format for nodes and properties.

More generally, I'd like to raise the bundle abstraction to a higher
level. As a first step I'd make the PersistenceManager interface aware
of bundles (i.e. load and store bundles instead of individual node and
property states), and later perhaps replace the item state concept
with bundle states to reduce the number of objects we need to keep
track of.

We might also consider not storing each bundle in a separate record,
instead making a single record of all the bundles included in a
ChangeLog instance. Such super-bundles could perhaps be persisted in
the data store.

> == Name and Namespace Index ==
> [...]
> == Commonly Used String ==

+1

BR,

Jukka Zitting