Fwd: Node failure due to Incremental repair

[Karthick V <ka...@gmail.com>]

Hi,
Recently I have enabled incremental repair in one of my test cluster setup
which consists of 8 nodes(DC1 - 4, DC2 - 4) with C* version of 2.1.13.
Currently, I am facing node failure scenario in this cluster with the
following exception during the incremental repair process

exception occurred during clean-up.  java.lang.reflect.
UndeclaredThrowableException
error: JMX connection closed. You should check server log for repair status
of keyspace VERTICALCRM(Subsequent keyspaces are not going to be repaired).
-- StackTrace --
java.io.IOException: JMX connection closed. You should check server log for
repair status of keyspace VERTICAL(Subsequent keyspaces are not going to be
repaired).
        at org.apache.cassandra.tools.RepairRunner.
handleNotification(NodeProbe.java:1496)
        at javax.management.NotificationBroadcasterSupport
.handleNotification(NotificationBroadcasterSupport.java:275)
        at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(
NotificationBroadcasterSupport.java:352)
        at javax.management.NotificationBroadcasterSupport$1.execute(
NotificationBroadcasterSupport.java:337)
        at javax.management.NotificationBroadcasterSupport.sendNotification(
NotificationBroadcasterSupport.java:248)
        at javax.management.remote.rmi.RMIConnector.sendNotification(
RMIConnector.java:441)
        at javax.management.remote.rmi.RMIConnector.access$1200(
RMIConnector.java:121)
        at javax.management.remote.rmi.RMIConnector$
RMIClientCommunicatorAdmin.gotIOException(RMIConnector.java:1531)
        at javax.management.remote.rmi.RMIConnector$RMINotifClient.
fetchNotifs(RMIConnector.java:1352)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$
NotifFetcher.fetchOneNotif(ClientNotifForwarder.java:655)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$
NotifFetcher.fetchNotifs(ClientNotifForwarder.java:607)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$
NotifFetcher.doRun(ClientNotifForwarder.java:471)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$
NotifFetcher.run(ClientNotifForwarder.java:452)
        at com.sun.jmx.remote.internal.ClientNotifForwarder$
LinearExecutor$1.run(ClientNotifForwarder.java:108)

And the node was made down by this exception.

When I tried to starting the same node I got following exception

java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<
init>(CompressedRandomAccessReader.java:73)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.
open(CompressedRandomAccessReader.java:48)
        at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile
.createPooledReader(CompressedPoolingSegmentedFile.java:95)
        at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(
PoolingSegmentedFile.java:62)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(
SSTableReader.java:1902)
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(
SimpleSliceReader.java:57)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.
createReader(SSTableSliceIterator.java:65)
        at org.apache.cassandra.db.columniterator.
SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
        at org.apache.cassandra.db.filter.SliceQueryFilter.
getSSTableColumnIterator(SliceQueryFilter.java:246)
        at org.apache.cassandra.db.filter.QueryFilter.
getSSTableColumnIterator(QueryFilter.java:62)
        at org.apache.cassandra.db.CollationController.collectAllData(
CollationController.java:270)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(
CollationController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(
ColumnFamilyStore.java:2001)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(
ColumnFamilyStore.java:1844)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:353)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(
SliceFromReadCommand.java:85)
        at org.apache.cassandra.cql3.statements.SelectStatement.
readLocally(SelectStatement.java:309)
        at org.apache.cassandra.cql3.statements.SelectStatement.
executeInternal(SelectStatement.java:328)
        at org.apache.cassandra.cql3.statements.SelectStatement.
executeInternal(SelectStatement.java:67)
        at org.apache.cassandra.cql3.QueryProcessor.executeInternal(
QueryProcessor.java:317)
        at org.apache.cassandra.db.SystemKeyspace.getSSTableReadMeter(
SystemKeyspace.java:972)
        at org.apache.cassandra.io.sstable.SSTableReader$
GlobalTidy.ensureReadMeter(SSTableReader.java:2388)
        at org.apache.cassandra.io.sstable.SSTableReader$
InstanceTidier.setup(SSTableReader.java:2204)
        at org.apache.cassandra.io.sstable.SSTableReader.setup(
SSTableReader.java:2145)
        at org.apache.cassandra.io.sstable.SSTableReader.open(
SSTableReader.java:491)
        at org.apache.cassandra.io.sstable.SSTableReader.open(
SSTableReader.java:384)
        at org.apache.cassandra.io.sstable.SSTableReader$4.run(
SSTableReader.java:531)
        at java.util.concurrent.Executors$RunnableAdapter.
call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

While debugging the issue, I started checking the SSTable count which had
been drastically increased ( say from 12 SStables to 2017 SStables in a
particular CF  ). Can anyone explain what goes wrong here? and is there any
possible way to start the node which is currently down?

P.S : I have followed all the *Recommeded Production settings* for this
cluster