You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kvrocks.apache.org by "advisedy (via GitHub)" <gi...@apache.org> on 2026/07/14 07:17:06 UTC
[PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
advisedy opened a new pull request, #3559:
URL: https://github.com/apache/kvrocks/pull/3559
Fixes #3100
## Problem
TSan reported an oversized allocation:
```
ThreadSanitizer: requested allocation size 0x100000000000000
exceeds maximum supported size of 0x10000000000
```
Root cause:
Before the fix, `GetFullReplDataInfo` released the lock before calling `GetChildren`, while `Server::cron` called `rocksdb::DestroyDB` on the live checkpoint directory without holding the lock at all. This allowed the following interleaving:
```
meta: unlock checkpoint_mu_
purge: DestroyDB starts deleting files
meta: GetChildren → [] (dir exists but files are already gone)
```
`GetFullReplDataInfo` builds the file list by appending entries and then calls `files->pop_back()`. If the directory is empty, `files` is an empty string, and `pop_back` on an empty string is undefined behavior. (https://en.cppreference.com/cpp/string/basic_string/pop_back)
On libstdc++, the common manifestation is a size underflow that produces a very large number. The subsequent `files + CRLF` then allocates memory based on this corrupted size, triggering the TSan error.
eaxmple:
```cpp
#include <cstdio>
#include <string>
int main() {
std::string files;
files.pop_back();
auto s = files + "\r\n";
printf("size=%zu\n", files.size());
return 0;
}
```
Running with GCC 11 + ASan on Linux:
```
AddressSanitizer: requested allocation size 0x100000000000000
(0x100000000001000 after adjustments for alignment, red zones etc.)
exceeds maximum supported size of 0x10000000000 (thread T0)
#0 0x7ffffee8b1e7 in operator new(unsigned long)
../../../../src/libsanitizer/asan/asan_new_delete.cpp:99
#1 0x7ffffecf8109 in void
std::__cxx11::basic_string::_M_construct<char*>(...)
(/lib/x86_64-linux-gnu/libstdc++.so.6+0x14f109)
...
SUMMARY: AddressSanitizer: allocation-size-too-big
```
The corrupted size (`0x100000000000000`) and the threshold (`0x10000000000`) match the TSan report in the issue.
While investigating, we also found that the same race condition, besides the "empty list" extreme case, could cause the replica to receive an incomplete file list (`GetChildren` returned only a subset of files).
---
## Fix
No `pop_back` on empty list:
In `GetFullReplDataInfo`, if `files` is empty, return `NotOK` immediately instead of calling `pop_back`.
Introduce `TryPurgeCheckpoint` — rename then destroy:
Added `Storage::TryPurgeCheckpoint`, which changes the purge logic to:
1. Under `checkpoint_mu_`, atomically rename `checkpoint_dir` to `checkpoint_dir.trash` via `RenameFile`;
2. Clear `checkpoint_info_`;
3. After releasing the lock, call `DestroyDB(trash)`.
This way, the meta listing either sees a complete live directory or no directory at all.
---
## Tests
Two new C++ gtests added (`storage_test.cc`):
- `GetFullReplDataInfoRejectsEmptyCheckpoint`: Creates a checkpoint → clears the directory → asserts `NotOK`.
- `TryPurgeCheckpointPolicyAndAtomicRemove`: Verifies idle purge, rename atomicity, and `checkpoint_info_` reset.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-4968731761
## [](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3559&metric=new_coverage&view=list) (required ≥ 50%)
<!-- sqra:qg_revision=27aa4b678e3c9290fcf388fb11b3286c04a84e78bb676021525bd6749e9c1e22 -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk commented on code in PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#discussion_r3608429128
##########
src/storage/storage.cc:
##########
@@ -1181,6 +1185,51 @@ bool Storage::ExistCheckpoint() {
bool Storage::ExistSyncCheckpoint() { return env_->FileExists(config_->sync_checkpoint_dir).ok(); }
+Status Storage::TryPurgeCheckpoint(int fetch_file_threads) {
+ std::string trash_dir = config_->checkpoint_dir + ".trash";
+ if (env_->FileExists(trash_dir).ok()) {
+ auto s = rocksdb::DestroyDB(trash_dir, rocksdb::Options());
+ if (!s.ok()) {
+ WARN("[storage] Fail to clean stale checkpoint trash, error: {}", s.ToString());
+ return {Status::NotOK, s.ToString()};
+ }
+ }
+
+ {
+ std::lock_guard<std::mutex> lg(checkpoint_mu_);
Review Comment:
We now have the condition check:`now_secs - access_time_secs > 30` to protect against this issue. And it should also have a low likelihood of being purged. For example, the time window between fetching metadata and files is more than 30s.
But the risk should be low, we can increase the time window to mitigate this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge [kvrocks]
Posted by "advisedy (via GitHub)" <gi...@apache.org>.
advisedy commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-5018003395
Thanks for the review! I'll wait for another reviewer to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge [kvrocks]
Posted by "advisedy (via GitHub)" <gi...@apache.org>.
advisedy commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-5018950414
Hi @git-hulk, thanks for triggering CI.
The failed Go integration job appears to match the known flake tracked in #3515:
--- FAIL: TestSlaveLostMaster (10.02s)
client.go:99: forwarding tcp stream stopped
client.go:147: forward tcp stream failed, err: read tcp 127.0.0.1:49530->127.0.0.1:49531: read: connection reset by peer
This seems unrelated to this PR. Could you please re-run the failed job when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge [kvrocks]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-5019382863
## [](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559) **Quality Gate failed**
Failed conditions
 [37.8% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3559&metric=new_coverage&view=list) (required ≥ 50%)
 [E Reliability Rating on New Code](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559) (required ≥ A)
<!-- sqra:qg_revision=a992f280b4a847e34fc48a9bebbcb49fa07dae90228984a1bf566a1ed068162b -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559)
##
💡 **Need a hand with PR review?** Try [Gitar by Sonar](https://gitar.ai/?utm_source=sonarqube-cloud&utm_medium=in-product&utm_campaign=gitar-crosssell-q3-2026&utm_content=pr-deco)!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-4967326164
## [](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559) **Quality Gate failed**
Failed conditions
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_kvrocks&pullRequest=3559&metric=new_coverage&view=list) (required ≥ 50%)
<!-- sqra:qg_revision=27aa4b678e3c9290fcf388fb11b3286c04a84e78bb676021525bd6749e9c1e22 -->
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_kvrocks&pullRequest=3559)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "advisedy (via GitHub)" <gi...@apache.org>.
advisedy commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-4967336134
Hi @jihuayu, thanks for the reminder. Here's my AI usage:
The main fix logic and code were written by myself.
I used AI for test and code review suggestions.
The additional test suggested by Copilot just now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#discussion_r3576939478
##########
tests/cppunit/storage_test.cc:
##########
@@ -170,3 +170,90 @@ TEST(Storage, ReplDataManagerRejectsUnsafeFilenames) {
std::filesystem::remove_all("test_repl_file_validation_dir", ec);
ASSERT_FALSE(ec);
}
+
+TEST(Storage, GetFullReplDataInfoRejectsEmptyCheckpoint) {
+ std::error_code ec;
+
+ const std::string test_dir = "test_empty_checkpoint";
+ Config config;
+ config.db_dir = test_dir + "/db";
+ config.checkpoint_dir = test_dir + "/checkpoint";
+ config.slot_id_encoded = false;
+
+ std::filesystem::remove_all(test_dir, ec);
+ ASSERT_FALSE(ec);
+ std::filesystem::create_directory(test_dir, ec);
+ ASSERT_FALSE(ec);
+
+ auto storage = std::make_unique<engine::Storage>(&config);
+ auto s = storage->Open();
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+
+ auto ctx = engine::Context(storage.get());
+ rocksdb::WriteBatch batch;
+ batch.Put("k", "v");
+ ASSERT_TRUE(storage->Write(ctx, rocksdb::WriteOptions(), &batch).ok());
+
+ std::string files;
+ s = engine::Storage::ReplDataManager::GetFullReplDataInfo(storage.get(), &files);
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+
+ std::filesystem::remove_all(config.checkpoint_dir, ec);
+ ASSERT_FALSE(ec);
+ std::filesystem::create_directory(config.checkpoint_dir, ec);
+ ASSERT_FALSE(ec);
+
+ files.clear();
+ s = engine::Storage::ReplDataManager::GetFullReplDataInfo(storage.get(), &files);
+ EXPECT_FALSE(s.IsOK());
+ EXPECT_TRUE(files.empty());
+
+ std::filesystem::remove_all(test_dir, ec);
+ ASSERT_FALSE(ec);
+}
+
+TEST(Storage, TryPurgeCheckpoint) {
+ std::error_code ec;
+
+ const std::string test_dir = "test_purge_checkpoint";
+ Config config;
+ config.db_dir = test_dir + "/db";
+ config.checkpoint_dir = test_dir + "/checkpoint";
+ config.slot_id_encoded = false;
+
+ std::filesystem::remove_all(test_dir, ec);
+ ASSERT_FALSE(ec);
+ std::filesystem::create_directory(test_dir, ec);
+ ASSERT_FALSE(ec);
+
+ auto storage = std::make_unique<engine::Storage>(&config);
+ auto s = storage->Open();
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+
+ auto ctx = engine::Context(storage.get());
+ rocksdb::WriteBatch batch;
+ batch.Put("k", "v");
+ ASSERT_TRUE(storage->Write(ctx, rocksdb::WriteOptions(), &batch).ok());
+
+ std::string files;
+ s = engine::Storage::ReplDataManager::GetFullReplDataInfo(storage.get(), &files);
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+
+ storage->SetCheckpointAccessTimeSecs(storage->GetCheckpointAccessTimeSecs() - 60);
+
+ s = storage->TryPurgeCheckpoint(/*fetch_file_threads=*/1);
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+ EXPECT_TRUE(storage->ExistCheckpoint());
+
+ s = storage->TryPurgeCheckpoint(/*fetch_file_threads=*/0);
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+ EXPECT_FALSE(storage->ExistCheckpoint());
+
+ files.clear();
+ s = engine::Storage::ReplDataManager::GetFullReplDataInfo(storage.get(), &files);
+ ASSERT_TRUE(s.IsOK()) << s.Msg();
+ EXPECT_TRUE(storage->ExistCheckpoint());
Review Comment:
This test exercises purge/no-purge paths, but it doesn’t actually assert the key invariants described in the PR (trash directory removed and checkpoint_info_ reset). Adding explicit assertions after the purge call would make the regression protection stronger and ensure TryPurgeCheckpoint clears state as intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge [kvrocks]
Posted by "git-hulk (via GitHub)" <gi...@apache.org>.
git-hulk merged PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on code in PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#discussion_r3591779586
##########
src/storage/storage.cc:
##########
@@ -1181,6 +1185,51 @@ bool Storage::ExistCheckpoint() {
bool Storage::ExistSyncCheckpoint() { return env_->FileExists(config_->sync_checkpoint_dir).ok(); }
+Status Storage::TryPurgeCheckpoint(int fetch_file_threads) {
+ std::string trash_dir = config_->checkpoint_dir + ".trash";
+ if (env_->FileExists(trash_dir).ok()) {
+ auto s = rocksdb::DestroyDB(trash_dir, rocksdb::Options());
+ if (!s.ok()) {
+ WARN("[storage] Fail to clean stale checkpoint trash, error: {}", s.ToString());
+ return {Status::NotOK, s.ToString()};
+ }
+ }
+
+ {
+ std::lock_guard<std::mutex> lg(checkpoint_mu_);
Review Comment:
With the current logic, the checkpoint can still be purged while it is in use because `GetFetchFileThreadNum()` is checked outside the lock.
The locking here does not seem sufficient. Could you take a look? @git-hulk @PragmaTwice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "jihuayu (via GitHub)" <gi...@apache.org>.
jihuayu commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-4966557570
Hello @advisedy. Thanks for your contribution. If you used an LLM, please follow our guidelines for AI-assisted contributions: https://kvrocks.apache.org/community/contributing#guidelines-for-ai-assisted-contributions
This does not mean that using an LLM is prohibited. It simply helps us assess the reliability of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(replication): prevent full-sync checkpoint race with cron purge (… [kvrocks]
Posted by "advisedy (via GitHub)" <gi...@apache.org>.
advisedy commented on PR #3559:
URL: https://github.com/apache/kvrocks/pull/3559#issuecomment-4977947586
Hi, @jihuayu . just friendly ping. CI has passed. Could you please take a look when you have time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@kvrocks.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org