You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefania (JIRA)" <ji...@apache.org> on 2016/04/07 09:44:25 UTC
[jira] [Commented] (CASSANDRA-11470) dtest failure in
materialized_views_test.TestMaterializedViews.base_replica_repair_test
[ https://issues.apache.org/jira/browse/CASSANDRA-11470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229849#comment-15229849 ]
Stefania commented on CASSANDRA-11470:
--------------------------------------
I fixed a problem in {{LogAwareFileLister.classifyFiles}}, which would have affected transactions spread across multiple disks. However, this is not what is causing these tests to fail unless we are using multiple disks and I haven't noticed.
I've also fixed another problem with the existing logic, in that we should only check that OLD sstable files still exist to determine if there is a race, we don't need to look at NEW sstable files. Since we cannot tell in advance how many NEW sstables files will be created and when, this might explain the problems we are seeing although I would have expected them on 3.0 as well.
Finally, if we fail to classify files due to an inconsistent disk state, I've added a log message showing the list of files detected and the content of the transaction.
Patch and CI results for trunk:
|[patch|https://github.com/stef1927/cassandra/commits/11470]|[testall|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-testall/]|[dtest|http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-11470-dtest/]|
Depending on the CI results, we may need to run {{base_replica_repair_test}} multiple times to see if the problem is fixed and to gather more information if it isn't.
> dtest failure in materialized_views_test.TestMaterializedViews.base_replica_repair_test
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-11470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11470
> Project: Cassandra
> Issue Type: Bug
> Reporter: Philip Thompson
> Assignee: Stefania
> Labels: dtest
> Fix For: 3.x
>
> Attachments: node1.log, node2.log, node2_debug.log, node3.log, node3_debug.log
>
>
> base_replica_repair_test has failed on trunk with the following exception in the log of node2:
> {code}
> ERROR [main] 2016-03-31 08:48:46,949 CassandraDaemon.java:708 - Exception encountered during startup
> java.lang.RuntimeException: Failed to list files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:53) ~[main/:na]
> at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:547) ~[main/:na]
> at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:725) ~[main/:na]
> at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:690) ~[main/:na]
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:567) ~[main/:na]
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:555) ~[main/:na]
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:383) ~[main/:na]
> at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:320) ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:130) ~[main/:na]
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:107) ~[main/:na]
> at org.apache.cassandra.cql3.restrictions.StatementRestrictions.<init>(StatementRestrictions.java:139) ~[main/:na]
> at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepareRestrictions(SelectStatement.java:864) ~[main/:na]
> at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:811) ~[main/:na]
> at org.apache.cassandra.cql3.statements.SelectStatement$RawStatement.prepare(SelectStatement.java:799) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.getStatement(QueryProcessor.java:505) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.parseStatement(QueryProcessor.java:242) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.prepareInternal(QueryProcessor.java:286) ~[main/:na]
> at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:294) ~[main/:na]
> at org.apache.cassandra.schema.SchemaKeyspace.query(SchemaKeyspace.java:1246) ~[main/:na]
> at org.apache.cassandra.schema.SchemaKeyspace.fetchKeyspacesWithout(SchemaKeyspace.java:875) ~[main/:na]
> at org.apache.cassandra.schema.SchemaKeyspace.fetchNonSystemKeyspaces(SchemaKeyspace.java:867) ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:134) ~[main/:na]
> at org.apache.cassandra.config.Schema.loadFromDisk(Schema.java:124) ~[main/:na]
> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:229) [main/:na]
> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) [main/:na]
> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:691) [main/:na]
> Caused by: java.lang.RuntimeException: Failed to list directory files in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985, inconsistent disk state for transaction [ma_txn_flush_58db56b0-f71d-11e5-bf68-03a01adb9f11.log in /mnt/tmp/dtest-du964e/test/node2/data0/system_schema/views-9786ac1cdd583201a7cdad556410c985]
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:149) ~[main/:na]
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:103) ~[main/:na]
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister$$Lambda$48/35984028.accept(Unknown Source) ~[na:na]
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) ~[na:1.8.0_45]
> at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_45]
> at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_45]
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:512) ~[na:1.8.0_45]
> at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:502) ~[na:1.8.0_45]
> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) ~[na:1.8.0_45]
> at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) ~[na:1.8.0_45]
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_45]
> at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) ~[na:1.8.0_45]
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:71) ~[main/:na]
> at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:49) ~[main/:na]
> ... 25 common frames omitted
> {code}
> example failure:
> http://cassci.datastax.com/job/trunk_dtest/1092/testReport/materialized_views_test/TestMaterializedViews/base_replica_repair_test
> Failed on CassCI build trunk_dtest #1092
> I've attached the logs from the failure in build 1092.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)