[jira] [Commented] (CASSANDRA-10697) Leak detected while running offline scrub

["Benedict (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15115066#comment-15115066 ] 

Benedict commented on CASSANDRA-10697:
--------------------------------------

I've pushed a fix for this [here|https://github.com/belliottsmith/cassandra/tree/10697-2.1]

It would be helpful if test engineering could have a look at the dtest that is meant to check this functionality, to try and get it to exercise this problem. I'm afraid my python is very rusty and I'm only here for a few more days.

In the process of merging this patch upwards, I found that it looks very much like scrub is not working at all in 2.2 upwards, as the transaction we are using doesn't get committed.  So the newly scrubbed sstables should be being deleted, and the old ones restored.  Could test engineering also check why the dtest isn't catching this problem?

> Leak detected while running offline scrub
> -----------------------------------------
>
>                 Key: CASSANDRA-10697
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10697
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: C* 2.1.9 on Debian Wheezy
>            Reporter: mlowicki
>            Assignee: Benedict
>            Priority: Critical
>
> I got couple of those:
> {code}
> ERROR 05:09:15 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@3b60e162) to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1433208674:/var/lib/cassandra/data/sync/entity2-e24b5040199b11e5a30f75bb514ae072/sync-entity2-ka-405434 was not released before the reference was garbage collected
> {code}
> and then:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:99)
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
>         at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:353)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
>         at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:378)
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348)
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
>         at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
>         at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:158)
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:220)
>         at org.apache.cassandra.tools.StandaloneScrubber.main(StandaloneScrubber.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)