You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Brandon Williams (Jira)" <ji...@apache.org> on 2022/10/14 15:51:00 UTC
[jira] [Commented] (CASSANDRA-17634) Fix test: dtest-upgrade.upgrade_tests.drop_compact_storage_upgrade_test.TestDropCompactStorage.test_drop_compact_storage_mixed_cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-17634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17617813#comment-17617813 ]
Brandon Williams commented on CASSANDRA-17634:
----------------------------------------------
bq. With the timeouts, where the test has many sleeps hence it's flakiness
bq. With sstable and CF mistracking. This may be a side effect of the above?
There are some best practices we've learned that this test isn't following in a few places, namely: waiting for schema agreement, and not waiting for the binary proto in once instance during upgrades. I think these can explain the above behaviors, modulo environmental timeouts, and [I have a patch for this|https://github.com/driftx/cassandra-dtest/tree/CASSANDRA-17634]. [!https://ci-cassandra.apache.org/job/Cassandra-devbranch/1999/badge/icon!|https://ci-cassandra.apache.org/blue/organizations/jenkins/Cassandra-devbranch/detail/Cassandra-devbranch/1999/pipeline]
In 4.1, there's another error:
bq. ERROR [NonPeriodicTasks:1] 2022-10-06 04:20:13,633 LogTransaction.java:398 - SSTableTidier ran with no existing data file for an sstable that was not new
Unfortunately, the [logs|https://nightlies.apache.org/cassandra/cassandra-4.1/Cassandra-4.1-dtest-upgrade/157/Cassandra-4.1-dtest-upgrade/label=cassandra-dtest,split=15/] are empty and this is the only run we have. Apparently there is another cause for the empty logs after CASSANDRA-17901 :(
> Fix test: dtest-upgrade.upgrade_tests.drop_compact_storage_upgrade_test.TestDropCompactStorage.test_drop_compact_storage_mixed_cluster
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-17634
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17634
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
> Reporter: Brandon Williams
> Assignee: Brandon Williams
> Priority: Normal
> Fix For: 4.0.x, 4.1.x
>
>
> From https://ci-cassandra.apache.org/job/Cassandra-4.1/24/testReport/dtest-upgrade.upgrade_tests.drop_compact_storage_upgrade_test/TestDropCompactStorage/test_drop_compact_storage_mixed_cluster/
> {noformat}
> Error Message
> test teardown failure
> Stacktrace
> Unexpected error found in node logs (see stdout for full details). Errors: [[node2] 'ERROR [NonPeriodicTasks:1] 2022-05-18 11:36:18,780 LogTransaction.java:398 - SSTableTidier ran with no existing data file for an sstable that was not new', [node2] 'ERROR [NonPeriodicTasks:1] 2022-05-18 11:36:18,788 LogTransaction.java:250 - Unable to delete /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-vk4cgziz/test/node2/data0/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_upgradesstables_bbe470a4-d69e-11ec-8d33-579a578037c7.log as it does not exist, see debug log file for stack trace', [node2] 'ERROR [NonPeriodicTasks:1] 2022-05-18 11:36:18,821 LogTransaction.java:250 - Unable to delete /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-vk4cgziz/test/node2/data0/system_schema/columns-24101c25a2ae3af787c1b40ee1aca33f/nb_txn_upgradesstables_bbe4709a-d69e-11ec-8d33-579a578037c7.log as it does not exist, see debug log file for stack trace', [node2] 'ERROR [CompactionExecutor:1] 2022-05-18 11:36:29,589 JVMStabilityInspector.java:68 - Exception in thread Thread[CompactionExecutor:1,5,CompactionExecutor]\njava.lang.NullPointerException: null\n\tat org.apache.cassandra.db.lifecycle.LogFile.makeRecord(LogFile.java:346)\n\tat org.apache.cassandra.db.lifecycle.LogFile.contains(LogFile.java:381)\n\tat org.apache.cassandra.db.lifecycle.LogTransaction.obsoleted(LogTransaction.java:183)\n\tat org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:136)\n\tat org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:206)\n\tat org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)\n\tat org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:386)\n\tat org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)\n\tat org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:106)\n\tat org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)\n\tat org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)\n\tat org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:116)\n\tat org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:218)\n\tat org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)\n\tat org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)\n\tat org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)\n\tat org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:299)\n\tat org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:98)\n\tat org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:47)\n\tat org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:57)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)']
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org