Re: Repairing OpsCenter rollups60 Results in Snapshot Errors

[Paul Nickerson <pg...@gmail.com>]

Thank you Reynald. I have contributed to that issue. But I cannot
participate further right now because now I'm having an out of memory issue
which may be unrelated. I think I'll start a new thread on this list for
that.


 ~ Paul Nickerson

On Thu, Jan 29, 2015 at 11:15 AM, Reynald Bourtembourg <
reynald.bourtembourg@esrf.fr> wrote:

>  Hi Paul,
>
> There is a JIRA ticket about this issue:
> https://issues.apache.org/jira/browse/CASSANDRA-8696
>
> I have seen these errors too the last time I ran "nodetool repair".
> I would also be interested to know the answer to the questions you were
> asking:
>
> "Are these errors problematic? Should I just let the repair process
> continue for however long it takes? "
> "I am wondering whether this is making the repair ineffectual."
>
> Best regards
>
> Reynald
>
>
>
> On 29/01/2015 17:03, Paul Nickerson wrote:
>
>  I am running a 6 node cluster using Apache Cassandra 2.1.2 with DataStax
> OpsCenter 5.0.2 from the AWS EC2 AMI "DataStax Auto-Clustering AMI
> 2.5.1-hvm" (DataStax Community AMI). When I try to run a repair on the
> rollups60 column family in the OpsCenter keyspace, I get errors about
> failed snapshot creation in the Cassandra system log. The repair seems to
> continue, and then finishes with errors.
>
>  I am wondering whether this is making the repair ineffectual.
>
>  I am running the command
>
>      nodetool repair OpsCenter rollups60
>
>  on one of the nodes (10.63.74.70). From the command, I get this output:
>
>      [2015-01-23 19:36:06,261] Starting repair command #9, repairing 511
> ranges for keyspace OpsCenter (seq=true, full=true)
>     [2015-01-23 21:08:16,242] Repair session
> 67772db0-a337-11e4-9e78-37e5027a626b for range
> (5848435723460298978,5868916338423419522] failed with error
> java.io.IOException: Failed during snapshot creation.
>
>  The error is repeated many times, and they all appear right at the end.
> Here is an example of what I see in the log on that same system (the one
> that I'm running the command from, and the one that's trying to snapshot):
>
>      INFO  [AntiEntropyStage:1] 2015-01-23 19:38:28,235
> RepairSession.java:171 - [repair #138b42e0-a337-11e4-9e78-37e5027a626b]
> Received merkle tree for rollups60 from /10.63.74.70
>     INFO  [AntiEntropySessions:9] 2015-01-23 19:38:28,236
> RepairSession.java:260 - [repair #67772db0-a337-11e4-9e78-37e5027a626b] new
> session: will sync /10.63.74.70, /10.51.180.16 on range
> (5848435723460298978,5868916338423419522] for OpsCenter.[rollups60]
>     INFO  [RepairJobTask:3] 2015-01-23 19:38:28,237 Differencer.java:74 -
> [repair #138b42e0-a337-11e4-9e78-37e5027a626b] Endpoints /10.13.157.190
> and /10.63.74.70 have 1 range(s) out of sync for rollups60
>     INFO  [AntiEntropyStage:1] 2015-01-23 19:38:28,237
> ColumnFamilyStore.java:840 - Enqueuing flush of rollups60: 465365 (0%)
> on-heap, 0 (0%) off-heap
>     INFO  [MemtableFlushWriter:25] 2015-01-23 19:38:28,238
> Memtable.java:325 - Writing Memtable-rollups60@204861223(51960 serialized
> bytes, 1395 ops, 0%/0% of on/off-heap limit)
>     INFO  [RepairJobTask:3] 2015-01-23 19:38:28,239
> StreamingRepairTask.java:68 - [streaming task
> #138b42e0-a337-11e4-9e78-37e5027a626b] Performing streaming repair of 1
> ranges with /10.13.157.190
>     INFO  [MemtableFlushWriter:25] 2015-01-23 19:38:28,262
> Memtable.java:364 - Completed flushing
> /raid0/cassandra/data/OpsCenter/rollups60-445613507ca411e4bd3f1927a2a71193/OpsCenter-rollups60-ka-331933-Data.db
> (29998 bytes) for commitlog position
> ReplayPosition(segmentId=1422038939094, position=31047766)
>     ERROR [RepairJobTask:2] 2015-01-23 19:38:39,067 RepairJob.java:127 -
> Error occurred during snapshot phase
>     java.lang.RuntimeException: Could not create snapshot at /10.63.74.70
>             at
> org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:77)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:347)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_51]
>             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_51]
>             at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_51]
>             at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_51]
>             at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
>     INFO  [AntiEntropySessions:10] 2015-01-23 19:38:39,068
> RepairSession.java:260 - [repair #6dec29c0-a337-11e4-9e78-37e5027a626b] new
> session: will sync /10.63.74.70, /10.51.180.16 on range
> (-6918744323658665195,-6916171087863528821] for OpsCenter.[rollups60]
>     ERROR [AntiEntropySessions:9] 2015-01-23 19:38:39,068
> RepairSession.java:303 - [repair #67772db0-a337-11e4-9e78-37e5027a626b]
> session completed with the following error
>     java.io.IOException: Failed during snapshot creation.
>             at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
>             at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [na:1.7.0_51]
>             at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_51]
>             at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
>     ERROR [AntiEntropySessions:9] 2015-01-23 19:38:39,070
> CassandraDaemon.java:153 - Exception in thread
> Thread[AntiEntropySessions:9,5,RMI Runtime]
>     java.lang.RuntimeException: java.io.IOException: Failed during
> snapshot creation.
>             at
> com.google.common.base.Throwables.propagate(Throwables.java:160)
> ~[guava-16.0.jar:na]
>             at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> ~[na:1.7.0_51]
>             at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> ~[na:1.7.0_51]
>             at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_51]
>             at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [na:1.7.0_51]
>             at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
>     Caused by: java.io.IOException: Failed during snapshot creation.
>             at
> org.apache.cassandra.repair.RepairSession.failedSnapshot(RepairSession.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> org.apache.cassandra.repair.RepairJob$2.onFailure(RepairJob.java:128)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
>             at
> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
>             ... 3 common frames omitted
>
>  The errors are repeated many times. The IP Address 10.63.74.70 in the
> log is the node I'm running the repair from. I am able to repair the rest
> of the OpsCenter column families, and they complete pretty quickly without
> error.
>
>  I have tried creating my own snapshot, and it completes successfully
> with nothing logged.
>
>      nodetool snapshot OpsCenter
>
>  The disk has plenty of space left. Are these errors problematic? Should
> I just let the repair process continue for however long it takes? The
> cluster is currently not in use by any application, yet it has some load
> while it's trying this repair, so it's not sitting idle (it has no load
> when I'm not repairing).
>
>  Thanks for any help.
>
>  And if this is the wrong place to ask about a DataStax Community thing,
> could someone point me in the right direction?
>
>   ~ Paul Nickerson
>
>
>