[jira] [Assigned] (CASSANDRA-7543) Assertion error when compacting large row with map//list field or range tombstone

["Brandon Williams (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CASSANDRA-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brandon Williams reassigned CASSANDRA-7543:
-------------------------------------------

    Assignee: Yuki Morishita

> Assertion error when compacting large row with map//list field or range tombstone
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7543
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7543
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: linux
>            Reporter: Matt Byrd
>            Assignee: Yuki Morishita
>              Labels: compaction, map
>
> Hi,
> So in a couple of clusters we're hitting this problem when compacting large rows with a schema which contains the map data-type.
> Here is an example of the error:
> {code}
> java.lang.AssertionError: incorrect row data size 87776427 written to /cassandra/X/Y/X-Y-tmp-ic-2381-Data.db; correct is 87845952
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:162)
> org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:163)
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58) 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
> {code}
> I have a python script which reproduces the problem, by just writing lots of data to a single partition key with a schema that contains the map data-type.
> I added some debug logging and found that the difference in bytes seen in the reproduction (255) was due to the following pieces of data being written:
> {code}
> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:42,891 ColumnIndex.java (line 168) DATASIZE writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: [java.nio.HeapByteBuffer[pos=0 lim=34 cap=34], java.nio.HeapByteBuffer[pos=0 lim=34 cap=34]](deletedAt=1405237116014999, localDeletion=1405237116) startPosition: 262476 endPosition: 262561 diff: 85 
> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:43,007 ColumnIndex.java (line 168) DATASIZE writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: org.apache.cassandra.db.Column@3e5b5939 startPosition: 328157 endPosition: 328242 diff: 85 
> DEBUG [CompactionExecutor:3] 2014-07-13 00:38:44,159 ColumnIndex.java (line 168) DATASIZE writeOpenedMarker columnIndex: org.apache.cassandra.db.ColumnIndex$Builder@6678a9d0 firstColumn: org.apache.cassandra.db.Column@fc3299b startPosition: 984105 endPosition: 984190 diff: 85
> {code}
> So looking at the code you can see that there are extra range tombstones written on the column index border (in ColumnIndex where tombstoneTracker.writeOpenedMarker is called) which aren't accounted for in LazilyCompactedRow.columnSerializedSize.
> This is where the difference comes from in the assertion error, so the solution is just to account for this data.
> I have a patch which does just this, by keeping track of the extra data written out via tombstoneTracker.writeOpenedMarker in ColumnIndex and adding it back to the dataSize in LazilyCompactedRow.java, where it serialises out the row size.
> After applying the patch the reproduction stops producing the AssertionError.
> I know this is not a problem in 2.0 + because of singe pass compaction, however there are lots of 1.2 clusters out there still which might run into this.
> Please let me know if you've any questions.
> Thanks,
> Matt



--
This message was sent by Atlassian JIRA
(v6.2#6252)