You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefania (JIRA)" <ji...@apache.org> on 2015/08/25 09:16:47 UTC
[jira] [Commented] (CASSANDRA-10159) Incorrect last update time
causes dtest to fail due to unexpected errors
[ https://issues.apache.org/jira/browse/CASSANDRA-10159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710761#comment-14710761 ]
Stefania commented on CASSANDRA-10159:
--------------------------------------
[~benedict] would you mind reviewing?
We should not check the last update time if there are no files. I've also increased the number of attempts when listing files, as Jenkins machines are a bit slower.
I've added a unit test that reproduces the problem and fixed some warnings in {{TransactionLogTest}}.
Patch: https://github.com/stef1927/cassandra/commits/10159-3.0
CI:
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10159-3.0-dtest/
http://cassci.datastax.com/view/Dev/view/stef1927/job/stef1927-10159-3.0-testall/
> Incorrect last update time causes dtest to fail due to unexpected errors
> ------------------------------------------------------------------------
>
> Key: CASSANDRA-10159
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10159
> Project: Cassandra
> Issue Type: Bug
> Reporter: Stefania
> Assignee: Stefania
> Fix For: 3.0.0 rc1
>
>
> Some dtests are failing as follows:
> http://cassci.datastax.com/job/cassandra-3.0_dtest/96/testReport/counter_tests/TestCounters/upgrade_test/
> {code}
> Unexpected error in node2 node log: ['ERROR [main] 2015-08-23 11:25:52,701 TransactionLog.java:246 - Possible disk corruption detected for sstable [ma-2-big], record [REMOVE:[ma-2-big,1440329048000,8]]: last update time [Thu Jan 01 00:00:00 UTC 1970] should have been [Sun Aug 23 11:24:08 UTC 2015] ERROR [main] 2015-08-23 11:25:52,709 TransactionLog.java:992 - Possible disk corruption: failed to read transaction log /mnt/tmp/dtest-E0OvQC/test/node2/data/system/local-7ad54392bcdd35a684174e047860b377/ma_txn_compaction_90eda9f0-4989-11e5-86bd-f32569933441.log org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) [main/:na] ERROR [main] 2015-08-23 11:25:52,710 TransactionLog.java:998 - Failed to remove unfinished transaction leftovers org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat
> {code}
> My best guess is that before reading the update time we should check that the file actually exists.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)