You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "David Capwell (Jira)" <ji...@apache.org> on 2021/06/28 18:26:00 UTC

[jira] [Commented] (CASSANDRA-16764) Compaction repeatedly fails validateReallocation exception

    [ https://issues.apache.org/jira/browse/CASSANDRA-16764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17370776#comment-17370776 ] 

David Capwell commented on CASSANDRA-16764:
-------------------------------------------

this is what I see in the code

{code}
static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
...
long validateReallocation(long newSize)
    {
        int saturatedSize = saturatedArraySizeCast(newSize);
        if (saturatedSize <= capacity())
            throw new RuntimeException();
        return saturatedSize;
    }
{code}

this is called by

{code}
long calculateNewSize(long count)
    {
        long capacity = capacity();
        //Both sides of this max expression need to use long arithmetic to avoid integer overflow
        //count and capacity are longs so that ensures it right now.
        long newSize = capacity + count;

        //For large buffers don't double, increase by 50%
        if (capacity > 1024L * 1024L * DOUBLING_THRESHOLD)
            newSize = Math.max((capacity * 3L) / 2L, newSize);
        else
            newSize = Math.max(capacity * 2L, newSize);

        return validateReallocation(newSize);
    }
{code}

which is called by

{code}
protected void expandToFit(long count)
    {
        if (count <= 0)
            return;
        ByteBuffer newBuffer = ByteBuffer.allocate(checkedArraySizeCast(calculateNewSize(count)));
        buffer.flip();
        newBuffer.put(buffer);
        buffer = newBuffer;
    }
{code}

This implies to me that the buffer is MAX_ARRAY_SIZE, at this point we are no longer able to expand (should return a better error in this case).  I say this as calculateNewSize should not be able to return a value < capacity, so saturatedArraySizeCast trimming to MAX_ARRAY_SIZE is what should trigger this.  If this is the case then it means the clustering key has a column larger than 
{code:java}
Integer.MAX_VALUE - 8
{code}
 bytes; what type is used for this table?


[~richardchesse] if you could take a look at a heap dump while this is happening it would help see what's going on.  If you have data ~ Integer.MAX_VALUE then I would expect other issues going on (such as GC issues).

> Compaction repeatedly fails validateReallocation exception
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-16764
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-16764
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local/Compaction/LCS
>            Reporter: Richard Hesse
>            Priority: Normal
>
> I have a few nodes in my ring that are stuck repeatedly trying to compact the same tables over and over again. I've run through the usual trick of rolling restarts, and it doesn't seem to help. This exception is logged on the nodes:
> {code}
> ERROR [CompactionExecutor:6] 2021-06-25 20:28:30,001 CassandraDaemon.java:244 - Exception in thread Thread[CompactionExecutor:6,1,main]
> java.lang.RuntimeException: null
>         at org.apache.cassandra.io.util.DataOutputBuffer.validateReallocation(DataOutputBuffer.java:134) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.util.DataOutputBuffer.calculateNewSize(DataOutputBuffer.java:152) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.util.DataOutputBuffer.expandToFit(DataOutputBuffer.java:159) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.util.DataOutputBuffer.doFlush(DataOutputBuffer.java:119) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:132) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.write(BufferedDataOutputStreamPlus.java:151) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.utils.ByteBufferUtil.writeWithVIntLength(ByteBufferUtil.java:296) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.marshal.AbstractType.writeValue(AbstractType.java:426) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.ClusteringPrefix$Serializer.serializeValuesWithoutSize(ClusteringPrefix.java:323) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.Clustering$Serializer.serialize(Clustering.java:131) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.ClusteringPrefix$Serializer.serialize(ClusteringPrefix.java:266) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:167) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.Serializers$NewFormatSerializer.serialize(Serializers.java:154) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:103) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.sstable.IndexInfo$Serializer.serialize(IndexInfo.java:82) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.ColumnIndex.addIndexBlock(ColumnIndex.java:216) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.ColumnIndex.add(ColumnIndex.java:264) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.ColumnIndex.buildRowIndex(ColumnIndex.java:111) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:173) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:136) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:98) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:143) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:204) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:272) ~[apache-cassandra-3.11.10.jar:3.11.10]
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292]
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
>         at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.10.jar:3.11.10]
>         at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_292]
> {code}
> Watching the compaction progress, it looks like it makes it through, dies, then starts over again. This process repeats forever.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org