[jira] [Comment Edited] (CASSANDRA-8211) Overlapping sstables in L1+

["Marcus Eriksson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-8211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14196034#comment-14196034 ] 

Marcus Eriksson edited comment on CASSANDRA-8211 at 11/4/14 12:05 PM:
----------------------------------------------------------------------

This was pretty hard to find (and reproduce), but I think i found the reason, see this example;

sstables after sstablesplit - a bunch of non-intersecting files it L0:
{code}
[ a ][ b ][ c ][ d ][ e ][ f ][ g ][ h ][ i ]
{code}
1. pick compaction candidates: a, i, start compacting (we pick the oldest sstables first)
2. pick compaction candidates: b, c, start compacting (note that none of these sstables intersect the compacting sstables)

Compaction 1 gives us, say (depending on LCS config), 3 sstables where the 'middle' one will cover basically the whole range. Then, when compaction 2 is done, we will end up with overlapping sstables in L1.

Solution is to make sure we never add an sstable that intersects with the boundaries of the compacting sstables. In the example above we have a compacting range of "the start of a" -> "the end of i" after step 1, meaning we could never start compaction nr2.

I've run this for a bunch of cycles now without seeing the errors.

The reason for us seeing this now is probably that people have started doing sstablesplit and running with no STCS compactions in L0 (the option from CASSANDRA-6621)


was (Author: krummas):
This was pretty hard to find (and reproduce), but I think i found the reason, see this example;

sstables after sstablesplit - a bunch of non-intersecting files:
{code}
[ a ][ b ][ c ][ d ][ e ][ f ][ g ][ h ][ i ]
{code}
1. pick compaction candidates: a, i, start compacting (we pick the oldest sstables first)
2. pick compaction candidates: b, c, start compacting (note that none of these sstables intersect the compacting sstables)

Compaction 1 gives us, say (depending on LCS config), 3 sstables where the 'middle' one will cover basically the whole range. Then, when compaction 2 is done, we will end up with overlapping sstables in L1.

Solution is to make sure we never add an sstable that intersects with the boundaries of the compacting sstables. In the example above we have a compacting range of "the start of a" -> "the end of i" after step 1, meaning we could never start compaction nr2.

I've run this for a bunch of cycles now without seeing the errors.

The reason for us seeing this now is probably that people have started doing sstablesplit and running with no STCS compactions in L0 (the option from CASSANDRA-6621)

> Overlapping sstables in L1+
> ---------------------------
>
>                 Key: CASSANDRA-8211
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8211
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Marcus Eriksson
>            Assignee: Marcus Eriksson
>             Fix For: 2.0.12
>
>         Attachments: 0001-Avoid-overlaps-in-L1.patch
>
>
> Seems we have a bug that can create overlapping sstables in L1:
> {code}
> WARN [main] 2014-10-28 04:09:42,295 LeveledManifest.java (line 164) At level 2, SSTableReader(path='<sstable>') [DecoratedKey(2838397575996053472, 00
> 10000000001111000000000000066059b200001000000000066059b200000000111100000000100000000000004000000000000000000100), DecoratedKey(5516674013223138308, 001000000000111100000000000000ff2d160000100000000000ff2d1600000
> 000111100000000100000000000004000000000000000000100)] overlaps SSTableReader(path='<sstable>') [DecoratedKey(2839992722300822584, 00100000000011110000
> 0000000000229ad20000100000000000229ad200000000111100000000100000000000004000000000000000000100), DecoratedKey(5532836928694021724, 0010000000001111000000000000034b05a600001000000000034b05a600000000111100000000100
> 000000000004000000000000000000100)].  This could be caused by a bug in Cassandra 1.1.0 .. 1.1.3 or due to the fact that you have dropped sstables from another node into the data directory. Sending back to L0.  If
>  you didn't drop in sstables, and have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable
> {code}
> Which might manifest itself during compaction with this exception:
> {code}
> ERROR [CompactionExecutor:3152] 2014-10-28 00:24:06,134 CassandraDaemon.java (line 199) Exception in thread Thread[CompactionExecutor:3152,1,main]
> java.lang.RuntimeException: Last written key DecoratedKey(5516674013223138308, 001000000000111100000000000000ff2d160000100000000000ff2d1600000000111100000000100000000000004000000000000000000100) >= current key DecoratedKey(2839992722300822584, 001000000000111100000000000000229ad20000100000000000229ad200000000111100000000100000000000004000000000000000000100) writing into <sstable>
> {code}
> since we use LeveledScanner when compacting (the backing sstable scanner might go beyond the start of the next sstable scanner)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)