You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Josh McKenzie (Jira)" <ji...@apache.org> on 2026/04/20 16:12:00 UTC
[jira] [Updated] (CASSANALYTICS-146) Remove StorageService shutdown hooks and clean up cluster re-use resource control
[ https://issues.apache.org/jira/browse/CASSANALYTICS-146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh McKenzie updated CASSANALYTICS-146:
----------------------------------------
Description:
When tests timeout the teardown on the in-jvm dtests races with StorageService’s shutdown hooks and lifecycle transaction files get double-deleted leading to thousands of lines of horrible useless log messages during failure.
This is particularly painful in the ecosystem integration tests as we stand up a lot of nodes and infra and it exposes us more to this kind of collision. The signal to noise ratio on timeout tests is brutal due to thousands of lines of "FSWriteError" on a bunch of lifecycle transaction /tmp files.
We should have no real risk of regression in the future since in tests we rely on forcing flushes in C* nodes to check sstables and other files, so the “are your memtables flushed on node shutdown during tests” question in the ecosystem shouldn’t ever come up."
Edit: While working on this it became clear the _root_ of this getting triggered is in the reusable cluster structure. There's a variety of things that aren't getting properly reset and cleaned up so a timeout in a single test will cascade to all class shared tests. We should fix that root cause as well as this noisy symptom together.
was:
When tests timeout the teardown on the in-jvm dtests races with StorageService’s shutdown hooks and lifecycle transaction files get double-deleted leading to thousands of lines of horrible useless log messages during failure.
This is particularly painful in the ecosystem integration tests as we stand up a lot of nodes and infra and it exposes us more to this kind of collision. The signal to noise ratio on timeout tests is brutal due to thousands of lines of "FSWriteError" on a bunch of lifecycle transaction /tmp files.
We should have no real risk of regression in the future since in tests we rely on forcing flushes in C* nodes to check sstables and other files, so the “are your memtables flushed on node shutdown during tests” question in the ecosystem shouldn’t ever come up."
> Remove StorageService shutdown hooks and clean up cluster re-use resource control
> ---------------------------------------------------------------------------------
>
> Key: CASSANALYTICS-146
> URL: https://issues.apache.org/jira/browse/CASSANALYTICS-146
> Project: Apache Cassandra Analytics
> Issue Type: Task
> Components: CI
> Reporter: Josh McKenzie
> Assignee: Josh McKenzie
> Priority: Normal
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When tests timeout the teardown on the in-jvm dtests races with StorageService’s shutdown hooks and lifecycle transaction files get double-deleted leading to thousands of lines of horrible useless log messages during failure.
> This is particularly painful in the ecosystem integration tests as we stand up a lot of nodes and infra and it exposes us more to this kind of collision. The signal to noise ratio on timeout tests is brutal due to thousands of lines of "FSWriteError" on a bunch of lifecycle transaction /tmp files.
> We should have no real risk of regression in the future since in tests we rely on forcing flushes in C* nodes to check sstables and other files, so the “are your memtables flushed on node shutdown during tests” question in the ecosystem shouldn’t ever come up."
> Edit: While working on this it became clear the _root_ of this getting triggered is in the reusable cluster structure. There's a variety of things that aren't getting properly reset and cleaned up so a timeout in a single test will cascade to all class shared tests. We should fix that root cause as well as this noisy symptom together.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org