[jira] [Created] (OAK-10758) Avoid or mitigate Cursor Timeout in Mongo when running detailedGC

["José Andrés Cordero Benítez (Jira)" <ji...@apache.org>]

José Andrés Cordero Benítez created OAK-10758:
-------------------------------------------------

             Summary: Avoid or mitigate Cursor Timeout in Mongo when running detailedGC
                 Key: OAK-10758
                 URL: https://issues.apache.org/jira/browse/OAK-10758
             Project: Jackrabbit Oak
          Issue Type: Improvement
          Components: documentmk, oak-run
            Reporter: José Andrés Cordero Benítez
            Assignee: José Andrés Cordero Benítez


When running oak-run revisions detailedGC command, if the cleanup is too big the processing time can exceed the Mongo cursor timeout, which defaults to 30 minutes:
{code:java}
java.io.IOException: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost' on server mongohost:27017
    at org.apache.jackrabbit.oak.run.RevisionsCommand.collectOnce(RevisionsCommand.java:427)
    at org.apache.jackrabbit.oak.run.RevisionsCommand.collect(RevisionsCommand.java:377)
    at org.apache.jackrabbit.oak.run.RevisionsCommand.execute(RevisionsCommand.java:245)
    at org.apache.jackrabbit.oak.run.Main.main(Main.java:49)
Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost:27017' on server mongohost:27017
    at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27)
    at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:268)
    at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:139)
    at com.mongodb.client.internal.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:54)
    at org.apache.jackrabbit.guava.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46)
    at org.apache.jackrabbit.guava.common.collect.Iterators$5.computeNext(Iterators.java:670)
    at org.apache.jackrabbit.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145)
    at org.apache.jackrabbit.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140)
    at org.apache.jackrabbit.guava.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395)
    at org.apache.jackrabbit.oak.plugins.document.SplitDocumentCleanUp.disconnect(SplitDocumentCleanUp.java:63)
    at org.apache.jackrabbit.oak.plugins.document.VersionGCSupport.deleteSplitDocuments(VersionGCSupport.java:146)
    at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.collectSplitDocuments(VersionGarbageCollector.java:842)
    at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.gc(VersionGarbageCollector.java:686)
    at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.run(VersionGarbageCollector.java:653)
    at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.gc(VersionGarbageCollector.java:221)
    at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:401)
    at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:398)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
15:45:08.184 [DocumentNodeStore cluster update thread (0)] WARN  o.a.j.o.p.document.DocumentNodeStore - Background operation failed: java.lang.reflect.UndeclaredThrowableException
java.lang.reflect.UndeclaredThrowableException: null
    at com.sun.proxy.$Proxy0.query(Unknown Source)
    at org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfoDocument.all(ClusterNodeInfoDocument.java:154)
    at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.updateClusterState(DocumentNodeStore.java:2480)
    at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundClusterUpdate.execute(DocumentNodeStore.java:3931)
    at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3734)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.reflect.InvocationTargetException: null
    at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.jackrabbit.oak.plugins.document.util.ReadOnlyDocumentStoreWrapperFactory$1.invoke(ReadOnlyDocumentStoreWrapperFactory.java:41)
    ... 6 common frames omitted
Caused by: java.lang.IllegalStateException: The pool is closed
    at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:137)
    at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:123)
    at com.mongodb.internal.session.ServerSessionPool.get(ServerSessionPool.java:80)
    at com.mongodb.internal.session.BaseClientSessionImpl.<init>(BaseClientSessionImpl.java:44)
    at com.mongodb.client.internal.ClientSessionImpl.<init>(ClientSessionImpl.java:56)
    at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:120)
    at com.mongodb.Mongo.createClientSession(Mongo.java:837)
    at com.mongodb.MongoClient.startSession(MongoClient.java:582)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoSessionFactory.createClientSession(MongoSessionFactory.java:55)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDBConnection.createClientSession(MongoDBConnection.java:115)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createClientSession(MongoDocumentStore.java:2405)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.execute(MongoDocumentStore.java:2389)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryInternal(MongoDocumentStore.java:846)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryWithRetry(MongoDocumentStore.java:774)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:751)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:739)
    at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:728)
    ... 10 common frames omitted {code}
Check if this needs mitigation or could happen as part of the internal repository detailedGC execution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)