Repository inconsistency problem (missing child)

[Jarosław Marek <ja...@gmail.com>]

Hi all,
I was asked to investigate an issue which occurred on our production
installation of Jackrabbit repo.
I was able to get whole repository packed and database schema
duplicated onto our development environment, so I could play with it
in any way I wanted. I was able to start the repository (on dev env)
without any errors in the logs. I'm attaching it's config in
repository_original.xml (you'll find more configuration details at the
end of this message).

Since the reported issue was only occurring on the default workspace,
I assumed that it is some kind of an inconsistency problem. I removed
all "index" directories to enforce re-indexing during repo startup and
then I got following error (full stacktrace attached: st1.txt):
RepositoryImpl.java:366 failed to start Repository: Unable to
initialize query handler:
org.apache.jackrabbit.core.query.lucene.SearchIndex@793536
which was caused by
org.apache.jackrabbit.core.state.NoSuchItemStateException:
67af1df0-127e-479d-8fe4-9651058de775

OK, at that point I knew something was missing but didn't know what
exactly. I googled and found the "onWorkspaceInconsistency"
configuration option, so I used it (setting value to "log") hoping to
get more info about missing node in the logs. I indeed got more info:
OnWorkspaceInconsistency.java:83 Node /
(cafebabe-cafe-babe-cafe-babecafebabe) has missing child 'hello'
(67af1df0-127e-479d-8fe4-9651058de775). Please run a consistency check
on this workspace!

So I started googling for consistency check and found info about
"enableConsistencyCheck" and "forceConsistencyCheck" parameters of
search index (http://jackrabbit.apache.org/frequently-asked-questions.html).
Happy that I found a solution I was suprised to see that this check
found nothing... After further googling I found this site:
http://documentation.magnolia-cms.com/display/DOCS/Repository+inconsistency
which mentions consistency check on workspaces.
Tried to use that, but it occured that we use an "old" Persistence
Manager (org.apache.jackrabbit.core.persistence.db.OraclePersistenceManager)
which does not support "consistencyCheck" or "consistencyFix"
parameters.
I was hoping that simply changing Persistence Manager class would do
the job, but "newer" Persistence Manager implementations use different
DB schema, so that couldn't work.

Knowing all these I though that I could try migrating the repository
into another one which would have newer PM configured so I could fix
my problem there. I set up a clean repository with a clean DB schema
(see attached repository_clean.xml), basically only difference in
config is that the new one uses
org.apache.jackrabbit.core.persistence.pool.OraclePersistenceManager
instead of db.OraclePersistenceManager.
I started new clean repo to see if it works and then shut it down.
Then I tried to migrate the old one using
jackrabbit-standalone-2.6.0.jar as described here:
http://jackrabbit.apache.org/standalone-server.html#StandaloneServer-Backupandmigration,
but got following error:
Exception in thread "main" javax.jcr.RepositoryException: Failed to copy content
at org.apache.jackrabbit.core.RepositoryCopier.copy(RepositoryCopier.java:211)
at org.apache.jackrabbit.core.RepositoryCopier.copy(RepositoryCopier.java:151)
at org.apache.jackrabbit.core.RepositoryCopier.copy(RepositoryCopier.java:112)
at org.apache.jackrabbit.standalone.Main.backup(Main.java:218)
at org.apache.jackrabbit.standalone.Main.run(Main.java:161)
at org.apache.jackrabbit.standalone.Main.main(Main.java:60)
Caused by: javax.jcr.RepositoryException: Unable to copy
67af1df0-127e-479d-8fe4-9651058de775
at org.apache.jackrabbit.core.persistence.PersistenceCopier.copy(PersistenceCopier.java:116)
at org.apache.jackrabbit.core.persistence.PersistenceCopier.copy(PersistenceCopier.java:110)
at org.apache.jackrabbit.core.RepositoryCopier.copyWorkspaces(RepositoryCopier.java:277)
at org.apache.jackrabbit.core.RepositoryCopier.copy(RepositoryCopier.java:209)
... 5 more
Caused by: org.apache.jackrabbit.core.state.NoSuchItemStateException:
67af1df0-127e-479d-8fe4-9651058de775
at org.apache.jackrabbit.core.persistence.db.DatabasePersistenceManager.load(DatabasePersistenceManager.java:379)
at org.apache.jackrabbit.core.persistence.PersistenceCopier.copy(PersistenceCopier.java:107)
... 8 more

Which was obviously caused by the missing node I had identified before.
After that I looked into the new repository (on file system level and
DB level) to see whether anything was migrated and it looks like it
was moved only partially. Most of the "non default" workspaces are
missing and those which seem to be present don't work (I couldn't
connect to them using Toromiro repository browser).

So now I'm stuck. I cannot migrate the repo because of the missing
node which I'm unable to fix without migrating the repo :/

I would appreciate any suggestions on how to fix missing child without
"consistencyFix". Is there anything I could manually delete from the
DB?

Thanks in advance,
Jarek

Configuration details:
Jackrabbit version: 2.6.0
deployed as a webapp (clients are connecting to it via WebDAV) in a
clustered configuration (two nodes on production, but only one being
active)
repository.xml files attached

Please let me know if you need any more details.