You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefania (JIRA)" <ji...@apache.org> on 2015/08/25 08:25:45 UTC
[jira] [Updated] (CASSANDRA-10159) Incorrect last update time
causes dtest to fail due to unexpected errors
[ https://issues.apache.org/jira/browse/CASSANDRA-10159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stefania updated CASSANDRA-10159:
---------------------------------
Description:
Some dtests are failing as follows:
http://cassci.datastax.com/job/cassandra-3.0_dtest/96/testReport/counter_tests/TestCounters/upgrade_test/
{code}
Unexpected error in node2 node log: ['ERROR [main] 2015-08-23 11:25:52,701 TransactionLog.java:246 - Possible disk corruption detected for sstable [ma-2-big], record [REMOVE:[ma-2-big,1440329048000,8]]: last update time [Thu Jan 01 00:00:00 UTC 1970] should have been [Sun Aug 23 11:24:08 UTC 2015] ERROR [main] 2015-08-23 11:25:52,709 TransactionLog.java:992 - Possible disk corruption: failed to read transaction log /mnt/tmp/dtest-E0OvQC/test/node2/data/system/local-7ad54392bcdd35a684174e047860b377/ma_txn_compaction_90eda9f0-4989-11e5-86bd-f32569933441.log org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) [main/:na] ERROR [main] 2015-08-23 11:25:52,710 TransactionLog.java:998 - Failed to remove unfinished transaction leftovers org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat
{code}
My best guess is that before reading the update time we should check that the file actually exists.
was:
Some dtests are failing as follows:
http://cassci.datastax.com/job/cassandra-3.0_dtest/lastCompletedBuild/testReport/counter_tests/TestCounters/upgrade_test/
{code}
Unexpected error in node2 node log: ['ERROR [main] 2015-08-23 11:25:52,701 TransactionLog.java:246 - Possible disk corruption detected for sstable [ma-2-big], record [REMOVE:[ma-2-big,1440329048000,8]]: last update time [Thu Jan 01 00:00:00 UTC 1970] should have been [Sun Aug 23 11:24:08 UTC 2015] ERROR [main] 2015-08-23 11:25:52,709 TransactionLog.java:992 - Possible disk corruption: failed to read transaction log /mnt/tmp/dtest-E0OvQC/test/node2/data/system/local-7ad54392bcdd35a684174e047860b377/ma_txn_compaction_90eda9f0-4989-11e5-86bd-f32569933441.log org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) [main/:na] ERROR [main] 2015-08-23 11:25:52,710 TransactionLog.java:998 - Failed to remove unfinished transaction leftovers org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat
{code}
My best guess is that before reading the update time we should check that the file actually exists.
> Incorrect last update time causes dtest to fail due to unexpected errors
> ------------------------------------------------------------------------
>
> Key: CASSANDRA-10159
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10159
> Project: Cassandra
> Issue Type: Bug
> Reporter: Stefania
> Assignee: Stefania
> Fix For: 3.0.0 rc1
>
>
> Some dtests are failing as follows:
> http://cassci.datastax.com/job/cassandra-3.0_dtest/96/testReport/counter_tests/TestCounters/upgrade_test/
> {code}
> Unexpected error in node2 node log: ['ERROR [main] 2015-08-23 11:25:52,701 TransactionLog.java:246 - Possible disk corruption detected for sstable [ma-2-big], record [REMOVE:[ma-2-big,1440329048000,8]]: last update time [Thu Jan 01 00:00:00 UTC 1970] should have been [Sun Aug 23 11:24:08 UTC 2015] ERROR [main] 2015-08-23 11:25:52,709 TransactionLog.java:992 - Possible disk corruption: failed to read transaction log /mnt/tmp/dtest-E0OvQC/test/node2/data/system/local-7ad54392bcdd35a684174e047860b377/ma_txn_compaction_90eda9f0-4989-11e5-86bd-f32569933441.log org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:622) [main/:na] ERROR [main] 2015-08-23 11:25:52,710 TransactionLog.java:998 - Failed to remove unfinished transaction leftovers org.apache.cassandra.db.lifecycle.TransactionLog$CorruptTransactionLogException: Failed to verify transaction 90eda9f0-4989-11e5-86bd-f32569933441 record [REMOVE:[ma-2-big,1440329048000,8]]: possible disk corruption, aborting \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionFile.readRecords(TransactionLog.java:349) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog$TransactionData.readLogFile(TransactionLog.java:574) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.TransactionLog.removeUnfinishedLeftovers(TransactionLog.java:988) ~[main/:na] \tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.removeUnfinishedLeftovers(LifecycleTransaction.java:548) [main/:na] \tat org.apache.cassandra.db.ColumnFamilyStore.scrubDataDirectories(ColumnFamilyStore.java:584) [main/:na] \tat org.apache.cassandra.service.StartupChecks$7.execute(StartupChecks.java:274) [main/:na] \tat org.apache.cassandra.service.StartupChecks.verify(StartupChecks.java:103) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:166) [main/:na] \tat org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [main/:na] \tat
> {code}
> My best guess is that before reading the update time we should check that the file actually exists.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)