You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "C. Scott Andreas (JIRA)" <ji...@apache.org> on 2018/11/19 02:09:00 UTC
[jira] [Resolved] (CASSANDRA-12762) Cassandra 3.0.9 Fails both
compact and repair without even debug logs
[ https://issues.apache.org/jira/browse/CASSANDRA-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
C. Scott Andreas resolved CASSANDRA-12762.
------------------------------------------
Resolution: Information Provided
> Cassandra 3.0.9 Fails both compact and repair without even debug logs
> ---------------------------------------------------------------------
>
> Key: CASSANDRA-12762
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12762
> Project: Cassandra
> Issue Type: Bug
> Environment: Debian Jessie current
> Reporter: Jason Kania
> Priority: Critical
>
> After upgrading from 3.0.7 to 3.0.9, the following exception occurs when trying to run compaction (previous to the upgrade compaction worked fine):
> {code}
> error: (/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db): corruption detected, chunk at 345885546 of length 62024.
> -- StackTrace --
> org.apache.cassandra.io.compress.CorruptBlockException: (/home/circuitwatch/cassandra/data/circuitwatch/edgeTransitionByCircuitId-f5d33310024b11e5bb310d2316086bf7/mb-12063-big-Data.db): corruption detected, chunk at 345885546 of length 62024.
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBufferMmap(CompressedRandomAccessReader.java:202)
> at org.apache.cassandra.io.util.RandomAccessReader.reBuffer(RandomAccessReader.java:111)
> at org.apache.cassandra.io.util.RebufferingInputStream.read(RebufferingInputStream.java:88)
> at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:66)
> at org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
> at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:404)
> at org.apache.cassandra.db.marshal.AbstractType.readValue(AbstractType.java:406)
> at org.apache.cassandra.db.rows.BufferCell$Serializer.deserialize(BufferCell.java:302)
> at org.apache.cassandra.db.rows.UnfilteredSerializer.readSimpleColumn(UnfilteredSerializer.java:476)
> at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeRowBody(UnfilteredSerializer.java:454)
> at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:377)
> at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87)
> at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65)
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100)
> at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30)
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95)
> at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32)
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189)
> at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158)
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:509)
> at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:369)
> at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47)
> at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
> at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111)
> at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52)
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:149)
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57)
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:183)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78)
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:606)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Seeing that there was a corruption, I ran nodetool repair, but it also fails in a spectacularly poor manner giving no indication of the problem:
> {code}
> error: Repair job has failed with the error message: [2016-10-08 01:25:22,431] Some repair failed
> -- StackTrace --
> java.lang.RuntimeException: Repair job has failed with the error message: [2016-10-08 01:25:22,431] Some repair failed
> at org.apache.cassandra.tools.RepairRunner.progress(RepairRunner.java:115)
> at org.apache.cassandra.utils.progress.jmx.JMXNotificationProgressListener.handleNotification(JMXNotificationProgressListener.java:77)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.dispatchNotification(ClientNotifForwarder.java:583)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.doRun(ClientNotifForwarder.java:533)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$NotifFetcher.run(ClientNotifForwarder.java:452)
> at com.sun.jmx.remote.internal.ClientNotifForwarder$LinearExecutor$1.run(ClientNotifForwarder.java:108)
> {code}
> The debug logs contain nothing and are just cut at the time of the error. Please update the errors in the repair case to communicate something meaningful as these sorts of errors mean the user is stranded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org