[jira] [Commented] (CASSANDRA-4799) assertion failure in leveled compaction test

["Yuki Morishita (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-4799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476329#comment-13476329 ] 

Yuki Morishita commented on CASSANDRA-4799:
-------------------------------------------

I found two problems here.

1) Test does not completely setup necessary SSTables and causes above AssertionError.

LCS test uses CFS#forceFlush to generate enough SSTables at the beginning of the test, but since the method is asynchronous call, there is a chance that test proceeds without sufficient SSTables to fill up L2 and causes AE at strat.getLevelSize(2) > 0.

Fix for this is to change forceFlush to forceBlockingFlush.
(diff: https://github.com/yukim/cassandra/commit/0d16efc6d7592e61f15598d5a4e3cc81d2760007)

2) Repair sometimes causes AssertionError with LCS

During the test, I encountered below error several times.

{code}
ERROR [ValidationExecutor:1] 2012-10-12 14:39:18,660 SchemaLoader.java (line 73) Fatal exception in thread Thread[ValidationExecutor:1,1,main]
java.lang.AssertionError
        at org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getScanners(LeveledCompactionStrategy.java:183)
        at org.apache.cassandra.db.compaction.CompactionManager$ValidationCompactionIterable.<init>(CompactionManager.java:879)
        at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:743)
        at org.apache.cassandra.db.compaction.CompactionManager.access$700(CompactionManager.java:71)
        at org.apache.cassandra.db.compaction.CompactionManager$7.call(CompactionManager.java:481)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
{code}

AssertionError comes from following assert code in LCS#getScanners:

{code}
for (SSTableReader sstable : sstables)
{
    int level = manifest.levelOf(sstable);
    assert level >= 0;
    byLevel.get(level).add(sstable);
}
{code}

LeveledManifest#levelOf method returns level of SSTable or -1 if SSTable is not yet added to LeveledManifest. Here, each SSTable comes from CF's DataTracker.
Every time SSTable is written by flush/compaction, it gets added to DataTracker and  after that, added to LeveledManifest if you are using LCS.
If above repair code is executed between those two, you will get AssertionError.

My proposed fix is to remove assertion and treat SSTables that do not belong LeveledManifest yet as L0 SSTables.
(diff: https://github.com/yukim/cassandra/commit/c8a0fb9a9128e47ec3d07926eb26f6fd93664f52)

Problem (2) also exists in 1.1 branch, but without assertion. We may want to fix this in 1.1 too.
                
> assertion failure in leveled compaction test
> --------------------------------------------
>
>                 Key: CASSANDRA-4799
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4799
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Brandon Williams
>            Assignee: Yuki Morishita
>             Fix For: 1.2.0
>
>
> It's somewhat rare, but I'm regularly seeing this failure on trunk:
> {noformat}
>     [junit] Testcase: testValidationMultipleSSTablePerLevel(org.apache.cassandra.db.compaction.LeveledCompactionStrategyTest):	FAILED
>     [junit] null
>     [junit] junit.framework.AssertionFailedError
>     [junit] 	at org.apache.cassandra.db.compaction.LeveledCompactionStrategyTest.testValidationMultipleSSTablePerLevel(LeveledCompactionStrategyTest.java:78)
>     [junit] 
>     [junit] 
>     [junit] Test org.apache.cassandra.db.compaction.LeveledCompactionStrategyTest FAILED
> {noformat}
> I suspect there's a deeper problem, since this is a pretty fundamental assertion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira