Consistency check of indexes takes to long

[Christoph Kiehl <ch...@sulu3000.de>]

Hi,

we've got very big indexes/workspaces on our production servers which have from 
3,000,000 to 8,000,000 nodes and are still growing because of creation of 
versions and adding new nodes.
When it happens that the VM in which Jackrabbit lives in crashes during a write 
operation, Jackrabbit nicely applies the redo log on a restart which gets done 
quite quick but then starts its consistency check. This check takes from 30 
minutes to 2 hours depending on the repository. In this time our application is 
offline which we would of course like to avoid ;) Our system uses a bundle 
oracle pm which probably doesn't make things better.
I had a quick glance at the consistency check code and it seems like there is 
nothing that could be substantially optimized in that place. I thought it might 
be possible to just include those index segments that where used while replaying 
the redo log but as the consistency check works this is impossible.
I think the only way to fasten startup is to avoid the occurrence of the errors 
that the check is checking for at all. Since the redo log mechanism seems quite 
good I'm not sure if those errors (MissingAncestor, MultipleEntries, 
NodeDeleted, UnknownParent) can still occur. Could you maybe elaborate on the 
situations where you expect those errors to arise?
For now I'm thinking about disabling consistency checks at all by default and 
run them in a maintenance window at night. Unfortunately this might be a bit 
dangerous if parts of the application rely on certain nodes to be found by 
queries :/
WDYT?

Cheers,
Christoph

Re: Consistency check of indexes takes to long

[Christoph Kiehl <ch...@sulu3000.de>]

Marcel Reutegger wrote:
> Christoph Kiehl wrote:
>> Well, I would prefer keeping the possible value set of "true" and 
>> "false" since the name of the value implies that it is a boolean flag. 
>> What about changing the the behaviour of Jackrabbit. If 
>> forceConsistencyCheck is "true" a check is run otherwise no check is 
>> run. AFAIU the consistency check is not necessarily dependent on the 
>> application of a redo log.
>> But I agree that either way the default behaviour should be to not do 
>> any consistency checks anymore.
> 
> glad to have you on board. you keep me away from introducing half-baked 
> changes into jackrabbit ;)
> 
> How about a new parameter:
> 
> enableConsistencyCheck = [true | false] (default: false)
> 
> or do you prefer:
> 
> disableConsistencyCheck = [true | false] (default: true)
> 
> I prefer the first one.

I would prefer the first one as well ;)

Cheers,
Christoph

Re: Consistency check of indexes takes to long

[Marcel Reutegger <ma...@gmx.net>]

Christoph Kiehl wrote:
> Well, I would prefer keeping the possible value set of "true" and 
> "false" since the name of the value implies that it is a boolean flag. 
> What about changing the the behaviour of Jackrabbit. If 
> forceConsistencyCheck is "true" a check is run otherwise no check is 
> run. AFAIU the consistency check is not necessarily dependent on the 
> application of a redo log.
> But I agree that either way the default behaviour should be to not do 
> any consistency checks anymore.

glad to have you on board. you keep me away from introducing half-baked changes 
into jackrabbit ;)

How about a new parameter:

enableConsistencyCheck = [true | false] (default: false)

or do you prefer:

disableConsistencyCheck = [true | false] (default: true)

I prefer the first one.

regards
  marcel

Re: Consistency check of indexes takes to long

[Christoph Kiehl <ch...@sulu3000.de>]

Marcel Reutegger wrote:

>> For now I'm thinking about disabling consistency checks at all by 
>> default and run them in a maintenance window at night. Unfortunately 
>> this might be a bit dangerous if parts of the application rely on 
>> certain nodes to be found by queries :/
>> WDYT?
> 
> I agree with you. We could introduce a third configuration value for 
> forceConsistenceCheck (in addition to 'true' and 'false'): disabled. 
> that would then be the default in the next released version of jackrabbit.

Well, I would prefer keeping the possible value set of "true" and "false" since 
the name of the value implies that it is a boolean flag. What about changing the 
the behaviour of Jackrabbit. If forceConsistencyCheck is "true" a check is run 
otherwise no check is run. AFAIU the consistency check is not necessarily 
dependent on the application of a redo log.
But I agree that either way the default behaviour should be to not do any 
consistency checks anymore.

Cheers,
Christoph

Re: Consistency check of indexes takes to long

[Marcel Reutegger <ma...@gmx.net>]

Hi Christoph,

Christoph Kiehl wrote:
> we've got very big indexes/workspaces on our production servers which 
> have from 3,000,000 to 8,000,000 nodes and are still growing because of 
> creation of versions and adding new nodes.
> When it happens that the VM in which Jackrabbit lives in crashes during 
> a write operation, Jackrabbit nicely applies the redo log on a restart 
> which gets done quite quick but then starts its consistency check. This 
> check takes from 30 minutes to 2 hours depending on the repository. In 
> this time our application is offline which we would of course like to 
> avoid ;) Our system uses a bundle oracle pm which probably doesn't make 
> things better.
> I had a quick glance at the consistency check code and it seems like 
> there is nothing that could be substantially optimized in that place. I 
> thought it might be possible to just include those index segments that 
> where used while replaying the redo log but as the consistency check 
> works this is impossible.
> I think the only way to fasten startup is to avoid the occurrence of the 
> errors that the check is checking for at all. Since the redo log 
> mechanism seems quite good I'm not sure if those errors 
> (MissingAncestor, MultipleEntries, NodeDeleted, UnknownParent) can still 
> occur. Could you maybe elaborate on the situations where you expect 
> those errors to arise?

IIRC the consistence check was introduced first in jackrabbit and later the redo 
log mechanism, which makes the consistence check kind of superfluous.

> For now I'm thinking about disabling consistency checks at all by 
> default and run them in a maintenance window at night. Unfortunately 
> this might be a bit dangerous if parts of the application rely on 
> certain nodes to be found by queries :/
> WDYT?

I agree with you. We could introduce a third configuration value for 
forceConsistenceCheck (in addition to 'true' and 'false'): disabled. that would 
then be the default in the next released version of jackrabbit.

WDYT?

regards
  marcel