You are viewing a plain text version of this content. The canonical link for it is here.
Posted to oak-dev@jackrabbit.apache.org by Alex Parvulescu <al...@gmail.com> on 2013/05/29 11:06:06 UTC
Asynchronous indexing consistency
hi guys,
I'm trying to find a solution for keeping the async index up to date in the
case where the system restarts.
How can the indexing process pick up where it left off?
A quick chat with Jukka turned up some ideas, but be warned this is all
based on api changes:
- expose the revision info and make it usable from the NodeStore.
The indexer would need to access the rev info, eventually store it, use it
as a reference when creating a new index branch.
- have a persistent (not garbage collected) branch dedicated to the async
indexer, which it can reuse even after a restart, eventually dispose of it
when the indexing process is done with.
This is an optimization of the entire index branch creation bits.
So, missing apis:
- revision info in the NodeState,
- possibility to get the NodeState given a revision, (so that we can have
the diff from rev to head)
- branch creation with a name, branch retrieval by name, branch delete by
name
thanks,
alex
Re: Asynchronous indexing consistency
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
After some thought and discussions about this, it seems to me that the
tree copy approach is probably too complex to implement. Instead I'd
like to propose a "checkpoint" mechanism that allows a client to
request a state of the repository to be excluded from garbage
collection for a specified amount of time.
The proposed new MicroKernel method would be:
/**
* Creates a new checkpoint of the latest head revision. The checkpoint
* guarantees that revision to remain valid and accessible for at least
* as long as requested.
*
* @param lifetime time (in milliseconds) that the checkpoint should
* remain available
* @return revision id of the created checkpoint
*/
String checkpoint(long lifetime);
And the mechanism could be reflected in the NodeStore interface like this:
/**
* Creates a new checkpoint of the latest root of the tree. The checkpoint
* remains valid for at least as long as requested and allows that state
* of the repository to be retrieved using the returned opaque string
* reference.
*
* @param lifetime time (in milliseconds) that the checkpoint should
* remain available
* @return string reference of this checkpoint
*/
String checkpoint(long lifetime);
/**
* Retrieves the root node from a previously created repository checkpoint.
*
* @param checkpoint string reference of a checkpoint
* @return the root node of the checkpoint,
* or {@code null} if the checkpoint is no longer available
*/
NodeState retrieve(String checkpoint);
BR,
Jukka Zitting
Re: Asynchronous indexing consistency
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>A complication here is the way the current index configurations nodes
>are structured, as we'd have to copy the root to something like
>/oak:index/someIndex/:before and without careful pre- or
>post-processing that would end up creating a recursive sequence of
>past revisions at
>/oak:index/someIndex/:before/oak:index/someIndex/:before/...
Hm, wouldn't we need to copy (just) all user defined content, excluding
index data? That means, if a user create a node under "/test" we would
need to copy that?
copy(source="/test", target="/<were we want the backup>/test")
But if we do this: the user could potentially create many child nodes in
the root node, and we would have to copy each one individually, each time
we want to create such a "backup". In theory that could be millions of
nodes.
One somewhat nice way to avoid that (well it has other problems) is if we
keep all user defined content (the root node of the repository) in a node
_below_ the root, for example in /jcr:root/. Or we could use the workspace
name :-) In any case that would be quite a big change.
A (rather ugly) alternative would be to support an "exclude list" for the
copy operation (so the source and target don't overlap).
Regards,
Thomas
Re: Asynchronous indexing consistency
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, May 29, 2013 at 5:26 PM, Thomas Mueller <mu...@adobe.com> wrote:
> What I meant by "reading old revisions" is that we need a way to read old
> revisions. You have suggested to use the copy operation to do that, which
> is fine; another solution is to not garbage collect a certain revision;
> this would require a new API (a way to define which revision to keep), and
> the node state API would need to be extended to support using revisions
> explicitly.
Right.
> I think using the copy operation would be preferable, if we find a simple
> way to support it in the MongoMK. Possibly the MongoMK could internally
> implement the copy operation (when copying a large tree) as keeping a
> pointer to a revision. That way we don't need a new API. It would slightly
> complicate the MongoMK, specially if we need a way to change the copy.
> Just supporting a fast read-only copy (a snapshot) should be feasible I
> think.
Yes, read-only should be good enough.
A complication here is the way the current index configurations nodes
are structured, as we'd have to copy the root to something like
/oak:index/someIndex/:before and without careful pre- or
post-processing that would end up creating a recursive sequence of
past revisions at
/oak:index/someIndex/:before/oak:index/someIndex/:before/...
BR,
Jukka Zitting
Re: Asynchronous indexing consistency
Posted by Ian Boston <ie...@tfd.co.uk>.
Hi,
I've been watching this thread, and the situation sounds quite simular to
issues I was seeing in pre production load testing a bit over a year ago. I
started with Solr and switched to elastic search.
ElasticSearch uses a write ahead log on every instance in its cluster to
address this issue. I have not heard of or had problems with instances
being randomly reprovisioned/lost/trashed/gone AWOL for a time, except
where over 60% of the instances have been lost in the same incident.
The wal enables acceptance of update operations far sooner than would
otherwise be possible, and the lack of binary replication via segments
greatly reduces the requirement for hard commits, which in turn greatly
reduces the latency from the update triggering event, to the item being
available in the index. Because of all that, provided tokenising
large content bodies is excluded from the main indexing pathway, I
found that the journal to record update events prior to acceptance by
elastic search, and I was able to place that acceptance inside a
transaction boundary.
I am not certain if you can use the same technique in Solr, as IIUC it
replicates on segment fragments, after insertion, although I think one of
the commercial Lucene companies might have done something along these
lines. IIRC there is some form of wal in the last Solr 4 release.
One last point, if you are coĊsidering using an observation journal, please
make it distributed. IMHO the shared central journal in JR2 was alway a
bottleneck in any cluster, and although a custom version using timestamps
partially eliminated cluster wide synchronisation it never broke free of
the resolution of time synchronisation.
HTH, if not ignore.
Ian
On Thursday, May 30, 2013, Thomas Mueller wrote:
> Hi,
>
> >For example during large batch imports or content
> >migrations it might be useful to be able to speed things up by
> >disabling things like full text indexing.
>
> OK, it's good to have a concrete use case.
>
> For a migration, the easiest solution might be to re-create the index at
> the end. For a large import to an already large repository that might not
> be good enough; but I wonder how common that would be.
>
> >Or it could be that an
> >external index server like Solr is down for maintenance or other
> >reasons.
>
> That's also a good use case.
>
> > Such cases would obviously lead to some loss of
> >functionality, but probably wouldn't be too troublesome if the
> >relevant indexers were able to automatically pick up from where they
> >left.
>
> Yes, that's true. Specially because you can't typically predict how long
> the outage would be (not because of Solr - it might be a hardware failure
> or so).
>
> >>It sounds like reading with old revisions.
> >
> >Not really; let me rephrase. What I'm suggesting is something like this:
>
> Yes, I understand that you have used the "copy" operation to pin an old
> revision. That's nice because it doesn't need a new API; it can be
> implemented using the already existing copy operation.
>
> What I meant by "reading old revisions" is that we need a way to read old
> revisions. You have suggested to use the copy operation to do that, which
> is fine; another solution is to not garbage collect a certain revision;
> this would require a new API (a way to define which revision to keep), and
> the node state API would need to be extended to support using revisions
> explicitly.
>
> I think using the copy operation would be preferable, if we find a simple
> way to support it in the MongoMK. Possibly the MongoMK could internally
> implement the copy operation (when copying a large tree) as keeping a
> pointer to a revision. That way we don't need a new API. It would slightly
> complicate the MongoMK, specially if we need a way to change the copy.
> Just supporting a fast read-only copy (a snapshot) should be feasible I
> think.
>
> Regards,
> Thomas
>
>
Re: Asynchronous indexing consistency
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>For example during large batch imports or content
>migrations it might be useful to be able to speed things up by
>disabling things like full text indexing.
OK, it's good to have a concrete use case.
For a migration, the easiest solution might be to re-create the index at
the end. For a large import to an already large repository that might not
be good enough; but I wonder how common that would be.
>Or it could be that an
>external index server like Solr is down for maintenance or other
>reasons.
That's also a good use case.
> Such cases would obviously lead to some loss of
>functionality, but probably wouldn't be too troublesome if the
>relevant indexers were able to automatically pick up from where they
>left.
Yes, that's true. Specially because you can't typically predict how long
the outage would be (not because of Solr - it might be a hardware failure
or so).
>>It sounds like reading with old revisions.
>
>Not really; let me rephrase. What I'm suggesting is something like this:
Yes, I understand that you have used the "copy" operation to pin an old
revision. That's nice because it doesn't need a new API; it can be
implemented using the already existing copy operation.
What I meant by "reading old revisions" is that we need a way to read old
revisions. You have suggested to use the copy operation to do that, which
is fine; another solution is to not garbage collect a certain revision;
this would require a new API (a way to define which revision to keep), and
the node state API would need to be extended to support using revisions
explicitly.
I think using the copy operation would be preferable, if we find a simple
way to support it in the MongoMK. Possibly the MongoMK could internally
implement the copy operation (when copying a large tree) as keeping a
pointer to a revision. That way we don't need a new API. It would slightly
complicate the MongoMK, specially if we need a way to change the copy.
Just supporting a fast read-only copy (a snapshot) should be feasible I
think.
Regards,
Thomas
Re: Asynchronous indexing consistency
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, May 29, 2013 at 3:01 PM, Thomas Mueller <mu...@adobe.com> wrote:
>>There could be various reasons for why an indexer might not be
>>available for an extended amount of time
>
> Possibly you are right, but let me try to challenge this assumption:
>
> Wouldn't it be a problem if the index isn't updated for a long time?
Not necessarily. For example during large batch imports or content
migrations it might be useful to be able to speed things up by
disabling things like full text indexing. Or it could be that an
external index server like Solr is down for maintenance or other
reasons. Such cases would obviously lead to some loss of
functionality, but probably wouldn't be too troublesome if the
relevant indexers were able to automatically pick up from where they
left.
> Don't we need a protection against an outdated index?
Any asynchronous indexes will in any case need to be resilient against
some mismatch between the index and repository content. Whether that
is measured in minutes or days should be irrelevant to the index
implementation, the only impact would be on the freshness assumptions
that applications or end users might have.
>>a) If, like in the Segment and H2 MKs, we could rely on the MKs
>>supporting cheap copies and diffs across subtrees, we could implement
>>this without API changes by keeping a copy of the last indexed/seen
>>state of the repository in a hidden subtree. The indexer would refresh
>>this copy on each index update, and could thus always know what
>>content has already been indexed. Unfortunately there probably isn't
>>any easy way to do this in the MongoMK.
>
> It sounds like reading with old revisions.
Not really; let me rephrase. What I'm suggesting is something like this:
NodeState root = branch.getHead();
NodeState index =
root.getChildNode("oak:index").getChildNode("someIndex");
NodeState before = index.getNode(":before");
NodeBuilder rootBuilder = root.builder();
NodeBuilder indexBuilder =
rootBuilder.getChildNode("oak:index").getChildNode("someIndex");
root.compareAgainstBaseState(
before, new IndexUpdate(indexBuilder.getChildNode(":index")));
indexBuilder.setChildNode(":before", root);
branch.setRoot(rootBuilder.getNodeState());
branch.merge();
I.e. instead of tracking things by revision, we'd just make a full
copy of the entire content tree that has already been indexed.
Unfortunately, AFAICT, in MongoMK this would require the duplication
of the entire subtree instead of the copy by reference that the
Segment and H2 MKs could do.
BR,
Jukka Zitting
Re: Asynchronous indexing consistency
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
>There could be various reasons for why an indexer might not be
>available for an extended amount of time
Possibly you are right, but let me try to challenge this assumption:
Wouldn't it be a problem if the index isn't updated for a long time? Don't
we need a protection against an outdated index? Unless each cluster node
keeps its own index, but I guess we should try to avoid that, for the sake
of scalability.
(I see there are other uses for an EventJournal, but it's easier if we
talk about concrete use cases).
>a) If, like in the Segment and H2 MKs, we could rely on the MKs
>supporting cheap copies and diffs across subtrees, we could implement
>this without API changes by keeping a copy of the last indexed/seen
>state of the repository in a hidden subtree. The indexer would refresh
>this copy on each index update, and could thus always know what
>content has already been indexed. Unfortunately there probably isn't
>any easy way to do this in the MongoMK.
It sounds like reading with old revisions. We would need a mechanism yet
to tell the MongoMK which revisions to keep; so far we have assumed a
fixed garbage collection limit (actually we didn't define that yet). The
MongoMK in fact it does currently keep all old revisions, because garbage
collection isn't implemented yet :-)
I see the difference to your approach is that intermediate revisions could
be garbage collected before, but that might be an unnecessary optimization.
Regards,
Thomas
Re: Asynchronous indexing consistency
Posted by Jukka Zitting <ju...@gmail.com>.
Hi,
On Wed, May 29, 2013 at 12:28 PM, Thomas Mueller <mu...@adobe.com> wrote:
> What would happen if you stop a cluster node for a long time (for example
> 1 day)? Would async indexing be done on another cluster node? If yes, I
> guess we need a way to ensure that's the case. If not, then the problem
> might be that old revisions are no longer available.
There could be various reasons for why an indexer might not be
available for an extended amount of time, so I think in any case we
need some mechanism for it to pick up from where it left. As you
mentioned, journaled observation will need some similar mechanism.
I see at least the following options:
a) If, like in the Segment and H2 MKs, we could rely on the MKs
supporting cheap copies and diffs across subtrees, we could implement
this without API changes by keeping a copy of the last indexed/seen
state of the repository in a hidden subtree. The indexer would refresh
this copy on each index update, and could thus always know what
content has already been indexed. Unfortunately there probably isn't
any easy way to do this in the MongoMK.
b) Have some way to mark specific revisions as ones that should be
kept around for a longer time (e.g. using a lease mechanism). The
indexer could then store such a revision id as a part of an index
update as a record of what content was last indexed.
c) Keep a log of all changes since the last index update. This is
probably the least attractive solution as it adds quite a bit of write
overhead and, unless the log is maintained by the MK, we'd still have
to worry about potential lost updates.
BR,
Jukka Zitting
Re: Asynchronous indexing consistency
Posted by Thomas Mueller <mu...@adobe.com>.
Hi,
What would happen if you stop a cluster node for a long time (for example
1 day)? Would async indexing be done on another cluster node? If yes, I
guess we need a way to ensure that's the case. If not, then the problem
might be that old revisions are no longer available.
I guess we need a similar mechanism for the JCR EventJournal. (Or we could
store the event journal as separate nodes, but that would need more space).
By the way, for the MongoMK, we need a mechanism to clean up garbage of
crashed cluster nodes. As we anyway need it, we might as well use it for
other features.
Regards,
Thomas
On 5/29/13 11:06 AM, "Alex Parvulescu" <al...@gmail.com> wrote:
>hi guys,
>
>I'm trying to find a solution for keeping the async index up to date in
>the
>case where the system restarts.
>How can the indexing process pick up where it left off?
>
>A quick chat with Jukka turned up some ideas, but be warned this is all
>based on api changes:
>
> - expose the revision info and make it usable from the NodeStore.
>The indexer would need to access the rev info, eventually store it, use it
>as a reference when creating a new index branch.
>
> - have a persistent (not garbage collected) branch dedicated to the async
>indexer, which it can reuse even after a restart, eventually dispose of it
>when the indexing process is done with.
>This is an optimization of the entire index branch creation bits.
>
>So, missing apis:
> - revision info in the NodeState,
> - possibility to get the NodeState given a revision, (so that we can have
>the diff from rev to head)
> - branch creation with a name, branch retrieval by name, branch delete by
>name
>
>
>thanks,
>alex