deadlock cleanup request due to compaction

["Desimpel, Ignace" <Ig...@nuance.com>]

After adding a 5-th node I started running NodeProbe:: forceKeyspaceCleanup. That function is not returning. Below I added the program stack trace information
that shows that in this case and probably others cases it is possible that there is a deadlock

In my case, the compactionManager is calling a synchronized method getNextBackgroundTask that is calling getMaximalTask. And in my case that function is running an endless loop so the synchronized object is never released. And the request to cleanup is now trying to call the synchronized pause() function, so that function and thus the cleanup cannot proceed.

Configuration : 5 node, v-nodes, replication 3, LCS, version 2.0.7



Daemon Thread [CompactionExecutor:72] (Suspended)
                owns: LeveledManifest  (id=8722)
                owns: LeveledCompactionStrategy  (id=8552)
                                waited by: Daemon System Thread [RMI TCP Connection(14840)-10.164.8.73] (Suspended)
                LeveledManifest.getCompactionCandidates() line: 247
                LeveledCompactionStrategy.getMaximalTask(int) line: 121
                LeveledCompactionStrategy.getNextBackgroundTask(int) line: 113
                CompactionManager$BackgroundCompactionTask.run() line: 191
                Executors$RunnableAdapter<T>.call() line: 471
                FutureTask<V>.run() line: 262
                CompactionManager$CompactionExecutor(ThreadPoolExecutor).runWorker(ThreadPoolExecutor$Worker) line: 1145
                ThreadPoolExecutor$Worker.run() line: 615
                Thread.run() line: 745


Daemon System Thread [RMI TCP Connection(14840)-10.164.8.73] (Suspended)
                owns: ColumnFamilyStore  (id=8551)
                waiting for: LeveledCompactionStrategy  (id=8552)
                                owned by: Daemon Thread [CompactionExecutor:72] (Suspended)
                LeveledCompactionStrategy(AbstractCompactionStrategy).pause() line: 112
                ColumnFamilyStore.runWithCompactionsDisabled(Callable<V>, boolean) line: 2056
                ColumnFamilyStore.markAllCompacting() line: 2125
                CompactionManager.performAllSSTableOperation(ColumnFamilyStore, CompactionManager$AllSSTablesOperation) line: 214
                CompactionManager.performCleanup(ColumnFamilyStore, CounterId$OneShotRenewer) line: 265
                ColumnFamilyStore.forceCleanup(CounterId$OneShotRenewer) line: 1105
                StorageService.forceKeyspaceCleanup(String, String...) line: 2215
                GeneratedMethodAccessor42.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                Trampoline.invoke(Method, Object, Object[]) line: 75
                GeneratedMethodAccessor19.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                MethodUtil.invoke(Method, Object, Object[]) line: 279
                StandardMBeanIntrospector.invokeM2(Method, Object, Object[], Object) line: 112
                StandardMBeanIntrospector.invokeM2(Object, Object, Object[], Object) line: 46
                StandardMBeanIntrospector(MBeanIntrospector<M>).invokeM(M, Object, Object[], Object) line: 237
                PerInterface<M>.invoke(Object, String, Object[], String[], Object) line: 138
                StandardMBeanSupport(MBeanSupport<M>).invoke(String, Object[], String[]) line: 252
                DefaultMBeanServerInterceptor.invoke(ObjectName, String, Object[], String[]) line: 819
                JmxMBeanServer.invoke(ObjectName, String, Object[], String[]) line: 801
                RMIConnectionImpl.doOperation(int, Object[]) line: 1487
                RMIConnectionImpl.access$300(RMIConnectionImpl, int, Object[]) line: 97
                RMIConnectionImpl$PrivilegedOperation.run() line: 1328
                RMIConnectionImpl.doPrivilegedOperation(int, Object[], Subject) line: 1420
                RMIConnectionImpl.invoke(ObjectName, String, MarshalledObject, String[], Subject) line: 848
                GeneratedMethodAccessor41.invoke(Object, Object[]) line: not available
                DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 43
                Method.invoke(Object, Object...) line: 606
                UnicastServerRef.dispatch(Remote, RemoteCall) line: 322
                Transport$1.run() line: 177
                Transport$1.run() line: 174
                AccessController.doPrivileged(PrivilegedExceptionAction<T>, AccessControlContext) line: not available [native method]
                TCPTransport(Transport).serviceCall(RemoteCall) line: 173
                TCPTransport.handleMessages(Connection, boolean) line: 556
                TCPTransport$ConnectionHandler.run0() line: 811
                TCPTransport$ConnectionHandler.run() line: 670
                ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: 1145
                ThreadPoolExecutor$Worker.run() line: 615
                Thread.run() line: 745