Revision stability with new MongoMK

[Jukka Zitting <ju...@gmail.com>]

Hi,

I gave a quick look at the new MongoMK prototype code, and was
wondering about the guarantees it gives on the stability of revisions.
More specifically, will the following assertion always hold?

    String r = mk.getHeadRevision();
    assert mk.getNodes("/", r, 0, 0, -1, null).equals(mk.getNodes("/",
r, 0, 0, -1, null));

In other words, can we expect a repository revision to remain stable over time?

The reason I'm asking is that it looks like two cluster nodes A and B
could be concurrently adding new nodes in a way that A finishes its
commit first, returning revisionA, and B following with revisionB. In
a case where B's clock is behind A's or B's cluster ID is smaller than
A's it would appear that revisionB came before revisionA, and thus if
the mk.getHeadRevision() returned revisionA it could be that the first
getNodes() call returned the state before revisionB was persisted and
the second one after that happened, resulting in the two returned JSON
strings being different.

Is this the intended behaviour? If yes, what needs to be changed
higher up in Oak to prevent this from causing problems (lost
observation events, missing index updates, etc.)?

BR,

Jukka Zitting

Re: Revision stability with new MongoMK

[Thomas Mueller <mu...@adobe.com>]

Hi,

>Do you have estimates on the potential performance impact of this?

I think in performance and memory overhead would be minimal, unless there
is a massive number of cluster nodes. The revision number consists of
[timestamp|machineId] (similar to the MongoDB ObjectId; machineId is the
same as the cluster node id). Based on that, revisions from a certain
cluster node can be ordered by timestamp+counter. When detecting revisions
(by reading the write counter at the root node and traversing the tree)
that were written by another cluster node, older revisions from the same
cluster node don't need to be kept in memory individually, as they can be
sorted by timestamp. Because of that, each cluster node wouldn't have to
maintain a complete list of all revisions, but only a list of revision
groups (only the exceptions to the timestamp|machineId order). For example:

[r1000|1]-[r2000|1], [r1200|2]-[r1300|2], [r2001|1]-[r3000|1]

In that case, the revisions of cluster node 1 ([r1000|1]-[r2000|1])
happened before revisions of cluster node 2 ([r1200|2]-[r1300|2]) but
before later revisions of cluster node 1 ([r2001|1]-[r3000|1]).

An open question is whether this complexity is really needed, or if we
could just do what databases usually do (read committed isolation level by
default, instead of repeatable read).

Regards,
Thomas

Re: Revision stability with new MongoMK

[Jukka Zitting <ju...@gmail.com>]

Hi,

On Wed, Mar 6, 2013 at 3:10 PM, Thomas Mueller <mu...@adobe.com> wrote:
> With the current implementation, that's not the case, because revision
> stability with multiple cluster nodes isn't implemented yet. The idea is
> that each cluster node adds revisions of other cluster nodes to its own
> revision list as they are detected.

I see, thanks! So cluster node A could see revisionA coming before
revisionB, while cluster node B might see the opposite, which should
be fine.

Do you have estimates on the potential performance impact of this? It
sounds like especially on a system with high volumes of writes the
revision lists could grow pretty large.

BR,

Jukka Zitting

Re: Revision stability with new MongoMK

[Thomas Mueller <mu...@adobe.com>]

Hi,

With the current implementation, that's not the case, because revision
stability with multiple cluster nodes isn't implemented yet. The idea is
that each cluster node adds revisions of other cluster nodes to its own
revision list as they are detected. Detection can be eager (when reading a
specific node) or done in a background thread. The background thread isn't
implemented yet as well. I understand the documentation on how this works
exactly is still missing. It simply wasn't a priority so far.

> what needs to be changed higher up in Oak

My plan (so far) was to make this invisible for the higher layers.

Regards,
Thomas



On 3/6/13 1:05 PM, "Jukka Zitting" <ju...@gmail.com> wrote:

>Hi,
>
>I gave a quick look at the new MongoMK prototype code, and was
>wondering about the guarantees it gives on the stability of revisions.
>More specifically, will the following assertion always hold?
>
>    String r = mk.getHeadRevision();
>    assert mk.getNodes("/", r, 0, 0, -1, null).equals(mk.getNodes("/",
>r, 0, 0, -1, null));
>
>In other words, can we expect a repository revision to remain stable over
>time?
>
>The reason I'm asking is that it looks like two cluster nodes A and B
>could be concurrently adding new nodes in a way that A finishes its
>commit first, returning revisionA, and B following with revisionB. In
>a case where B's clock is behind A's or B's cluster ID is smaller than
>A's it would appear that revisionB came before revisionA, and thus if
>the mk.getHeadRevision() returned revisionA it could be that the first
>getNodes() call returned the state before revisionB was persisted and
>the second one after that happened, resulting in the two returned JSON
>strings being different.
>
>Is this the intended behaviour? If yes, what needs to be changed
>higher up in Oak to prevent this from causing problems (lost
>observation events, missing index updates, etc.)?
>
>BR,
>
>Jukka Zitting