You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Miller (Jira)" <ji...@apache.org> on 2025/08/19 15:03:00 UTC
[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception
[ https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18014933#comment-18014933 ]
Chris Miller commented on CASSANDRA-16764:
------------------------------------------
Hi [~brandon.williams] [~dcapwell] - revisiting this issue as we have another occurrence in our prod environment which is resulting in OOM's. Any additional ideas on how to resolve / work around this? I still have the problem SSTABLEs available for testing if you have any suggestions.
Thanks,
Chris.
> Compaction repeatedly fails validateReallocation exception
> ----------------------------------------------------------
>
> Key: CASSANDRA-16764
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Local/Compaction/LCS
> Reporter: Richard Hesse
> Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the same tables over and over again. I've run through the usual trick of rolling restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244 - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
> at org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:272) ~[apache-cassandra-3.11.10.jar:3.11.10]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.10.jar:3.11.10]
> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
> {code}
> Watching the compaction progress, it looks like it makes it through, dies, then starts over again. This process repeats forever.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org