You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Prakash Chauhan <pr...@ericsson.com> on 2016/06/28 10:55:26 UTC

Exception in logs using LCS .

Hello,

Recently we changed compaction strategy fora table to LCS from the default STCS. While bootstrapping a node , we are getting following Exception in the logs:

ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key DecoratedKey(-8631371593982690738, 623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java (line 795) Enqueuing flush of Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1 ops)

Questions:

1.       Are these exceptions due to data corruption ?

2.       I am unable to reproduce the problem. How can I reproduce the Exception? Is there any specific case when such exceptions are raised?

3.       Without reproducing the Exception, how can I test the patch available at related JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935



Thanks,
Prakash Chauhan.

RE: Exception in logs using LCS .

Posted by Prakash Chauhan <pr...@ericsson.com>.
Hi  Paulo,

Thanks for the reply. Running scrub every time the exception occurs is not a possible solution for us. We don’t have a log patrolling system in place that verify the log on regular basis.

I have some queries based on your reply:

1.       You mentioned race condition. Can you please elaborate a little more what actual race condition may be happening in this scenario ?

2.       What if this happens in production ? What would be the impact? Can we live with it?

3.       Test system where the problem was reported has been scrapped so we can’t run user defined compaction now. Any suggestions to reproduce it on a new system?



From: Paulo Motta [mailto:pauloricardomg@gmail.com]
Sent: Tuesday, June 28, 2016 5:43 PM
To: user@cassandra.apache.org
Subject: Re: Exception in logs using LCS .

1. Not necessarily data corruption, but it seems compaction is trying to write data in the wrong order most likely due to a temporary race condition/bug a la #9935, but since the compaction fails your original data is probably safe (you can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which sstables were picked for compaction at a particular instant, but you could try running a user-defined compaction or scrub on the sstables that contain this key, see CASSANDRA-11337and https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the generated SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan <pr...@ericsson.com>>:
Hello,

Recently we changed compaction strategy fora table to LCS from the default STCS. While bootstrapping a node , we are getting following Exception in the logs:

ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java (line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
java.lang.RuntimeException: Last written key DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key DecoratedKey(-8631371593982690738, 623437393a313a30) writing into /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
        at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
        at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957 ColumnFamilyStore.java (line 795) Enqueuing flush of Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1 ops)

Questions:

1.       Are these exceptions due to data corruption ?

2.       I am unable to reproduce the problem. How can I reproduce the Exception? Is there any specific case when such exceptions are raised?

3.       Without reproducing the Exception, how can I test the patch available at related JIRA : https://issues.apache.org/jira/browse/CASSANDRA-9935



Thanks,
Prakash Chauhan.

Re: Exception in logs using LCS .

Posted by Paulo Motta <pa...@gmail.com>.
1. Not necessarily data corruption, but it seems compaction is trying to
write data in the wrong order most likely due to a temporary race
condition/bug a la #9935, but since the compaction fails your original data
is probably safe (you can try running scrub to verify/fix corruptions).
2. This is pretty tricky to reproduce because it will depend on which
sstables were picked for compaction at a particular instant, but you could
try running a user-defined compaction or scrub on the sstables that contain
this key, see CASSANDRA-11337and
https://gist.github.com/jeromatron/e238e5795b3e79866b83
3. clone the cassandra repository of your current version, git-cherry-pick
the commit of CASSANDRA-9935, ant jar, replace the cassandra jar with the
generated SNAPSHOT.jar and restart the node`

2016-06-28 7:55 GMT-03:00 Prakash Chauhan <pr...@ericsson.com>:

> Hello,
>
>
>
> Recently we changed compaction strategy fora table to LCS from the default
> STCS. While bootstrapping a node , we are getting following Exception in
> the logs:
>
>
>
> ERROR [CompactionExecutor:81] 2016-05-11 13:48:54,694 CassandraDaemon.java
> (line 258) Exception in thread Thread[CompactionExecutor:81,1,main]
>
> java.lang.RuntimeException: Last written key
> DecoratedKey(-2711050270696519088, 623330333a313a35) >= current key
> DecoratedKey(-8631371593982690738, 623437393a313a30) writing into
> /cassandra/data/main/myCF/myCF-tmp-jb-326-Data.db
>
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:143)
>
>         at
> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:166)
>
>         at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:167)
>
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
>
>         at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:60)
>
>         at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
>
>         at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:198)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
>         at java.lang.Thread.run(Thread.java:745)
>
> INFO [CompactionExecutor:68] 2016-05-11 14:10:28,957
> ColumnFamilyStore.java (line 795) Enqueuing flush of
> Memtable-compactions_in_progress@541886922(0/0 serialized/live bytes, 1
> ops)
>
>
>
> Questions:
>
> *1.       *Are these exceptions due to data corruption ?
>
> *2.       *I am unable to reproduce the problem. How can I reproduce the
> Exception? Is there any specific case when such exceptions are raised?
>
> *3.       *Without reproducing the Exception, how can I test the patch
> available at related JIRA :
> https://issues.apache.org/jira/browse/CASSANDRA-9935
>
>
>
>
>
> Thanks,
>
> Prakash Chauhan.
>
>
>
>
>
>
>
>
>
>
>