You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Richard Hesse (Jira)" <ji...@apache.org> on 2021/08/10 15:36:00 UTC
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails
validateReallocation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17396746#comment-17396746 ]
Richard Hesse commented on CASSANDRA-16764:
-------------------------------------------
Updating the bug. We had some success moving those data files related to large (~1.9GB) index files out of the way. Those are the large files around that max file size. Still playing whack-a-mole as repairs can finally run and more compactions roll through and create new, up-to-2GB index files. But overall I think we know our path forward. Thanks for everyone's help!
> Compaction repeatedly fails validateReallocation exception
> ----------------------------------------------------------
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
> Reporter: Richard Hesse
> Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the same tables over and over again. I've run through the usual trick of rolling restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244 - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:272) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.10.jar:3.11.10]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
> {code}
> Watching the compaction progress, it looks like it makes it through, dies, then starts over again. This process repeats forever.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org