Data corruption knowledge sharing

[Unico Homme <un...@apache.org>]

Hi all,

As you may have noticed I have been working on trying to improve the
consistency checker. There are quite some improvements I want to make and I
will try as much as possible to create separate issues for them to maximise
the transparency of what we want to achieve in this area. Until now we had
our own checker that is faster, checks for more different cases of
inconsistency, and fixes more cases of inconsistency than the current
checker in Jackrabbit. We'd like to donate and incorporate the knowledge
and experience that is reflected in our own checker to the JR checker.

As a parallel to that work I'd like to initiate a knowledge sharing
discussion about the possible causes of data corruption in Jackrabbit. At
Hippo we experience data corruption on a regular basis. And although we
have spent quite some time analysing the possible causes of this corruption
we've not been able until now to minimize it to our satisfaction.

Questions such as,

1. what are the scenario's in which data corruption is most likely to occur
2. what are the best strategies to minimize the chance of data corruption
in an application on top of JR
3. is the main cause of data corruption concurrrent writes on multiple
cluster nodes
4. or have you also seen significant problems in single cluster set ups
5. do you have ideas or suggestions how JR itself can be improved to be
more robust against data corruption

One of the scenario's we've experienced that causes regular data corruption
is concurrent add node operations on the same folder from different cluster
nodes.

For one use case in which such collisions can occur regularly we've
implemented a strategy to avoid this corruption using separate folders for
different cluster nodes. Off course this course is not always an option but
in certain scenario's it seems to be a good solution.

If you have similar experiences and/or know of other typical scenarios in
which data corruption can occur I'd love to hear about them.

Best regards,
Unico Hommes
Hippo BV, the Netherlands