You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2025/11/03 12:16:46 UTC
[PR] [refactor](filecache) ttl management refactoring [doris]
freemandealer opened a new pull request, #57647:
URL: https://github.com/apache/doris/pull/57647
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494376597
##########
be/src/io/cache/block_file_cache.cpp:
##########
@@ -717,6 +636,7 @@ FileBlocks BlockFileCache::split_range_into_cells(const UInt128Wrapper& hash,
cell->update_atime();
}
}
+ _ttl_mgr->register_tablet_id(context.tablet_id);
Review Comment:
从 tablet meta 补充进来
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #57647:
URL: https://github.com/apache/doris/pull/57647#issuecomment-3480236561
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2492832775
##########
be/src/io/cache/block_file_cache_ttl_mgr.cpp:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/block_file_cache_ttl_mgr.h"
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <thread>
+#include <unordered_set>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/cache_block_meta_store.h"
+#include "io/cache/file_block.h"
+#include "util/time.h"
+
+namespace doris::io {
+
+BlockFileCacheTtlMgr::BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store)
+ : _mgr(mgr), _meta_store(meta_store), _stop_background(false) {
+ // Start background threads
+ _update_ttl_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map, this);
+ _expiration_check_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_expiration_check, this);
+}
+
+BlockFileCacheTtlMgr::~BlockFileCacheTtlMgr() {
+ _stop_background.store(true, std::memory_order_release);
+
+ if (_update_ttl_thread.joinable()) {
+ _update_ttl_thread.join();
+ }
+
+ if (_expiration_check_thread.joinable()) {
+ _expiration_check_thread.join();
+ }
+}
+
+void BlockFileCacheTtlMgr::register_tablet_id(int64_t tablet_id) {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ _tablet_id_set.insert(tablet_id);
+}
+
+FileBlocks BlockFileCacheTtlMgr::get_file_blocks_from_tablet_id(int64_t tablet_id) {
+ FileBlocks result;
+
+ // Use meta store to get all blocks for this tablet
+ auto iterator = _meta_store->range_get(tablet_id);
+ if (!iterator) {
+ LOG(WARNING) << "Failed to get iterator for tablet_id: " << tablet_id;
+ return result;
+ }
+
+ while (iterator->valid()) {
+ BlockMetaKey key = iterator->key();
+ BlockMeta meta = iterator->value();
+
+ // Get all blocks for this hash using get_blocks_by_key
+ try {
+ auto blocks_map = _mgr->get_blocks_by_key(key.hash);
+ for (const auto& [offset, block] : blocks_map) {
+ // Only add blocks that match our specific offset
+ if (offset == key.offset) {
+ result.push_back(block);
+ break;
+ }
+ }
+ } catch (const std::exception& e) {
+ LOG(WARNING) << "Failed to get file blocks for tablet_id: " << tablet_id
+ << ", hash: " << key.hash.to_string() << ", error: " << e.what();
+ }
+
+ iterator->next();
+ }
+
+ return result;
+}
+
+void BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map() {
+ Thread::set_self_name("ttl_mgr_update");
+
+ while (!_stop_background.load(std::memory_order_acquire)) {
+ try {
+ std::unordered_set<int64_t> tablet_ids_to_process;
+
+ // Copy tablet IDs to process
+ {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ tablet_ids_to_process = _tablet_id_set;
+ }
+
+ for (int64_t tablet_id : tablet_ids_to_process) {
+ // TODO(zhengyu): Implement actual CloudMetaMgr or CloudTabletMgr integration
+ // For now, we'll use placeholder values
+ uint64_t create_time = 0;
+ uint64_t ttl = 0;
+
+ // Simulate getting tablet metadata
+ // This should be replaced with actual CloudMetaMgr::get_tablet_meta call
+ bool has_ttl = (tablet_id % 10 == 0); // Example condition
+ if (has_ttl) {
+ create_time = UnixSeconds(); // Current time as create time
+ ttl = 3600; // 1 hour TTL
+ }
+
+ // Update TTL info map
+ {
+ std::lock_guard<std::mutex> lock(_ttl_info_mutex);
+ if (ttl > 0) {
+ _ttl_info_map[tablet_id] = TtlInfo {ttl, create_time};
+
+ // If TTL changed from 0 to non-zero, convert blocks to TTL type
+ auto old_info_it = _ttl_info_map.find(tablet_id);
+ if (old_info_it == _ttl_info_map.end() || old_info_it->second.ttl == 0) {
+ FileBlocks blocks = get_file_blocks_from_tablet_id(tablet_id);
Review Comment:
need flow control to avoid long lock
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494470594
##########
be/src/io/cache/block_file_cache_ttl_mgr.cpp:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/block_file_cache_ttl_mgr.h"
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <thread>
+#include <unordered_set>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/cache_block_meta_store.h"
+#include "io/cache/file_block.h"
+#include "util/time.h"
+
+namespace doris::io {
+
+BlockFileCacheTtlMgr::BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store)
+ : _mgr(mgr), _meta_store(meta_store), _stop_background(false) {
+ // Start background threads
+ _update_ttl_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map, this);
+ _expiration_check_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_expiration_check, this);
+}
+
+BlockFileCacheTtlMgr::~BlockFileCacheTtlMgr() {
+ _stop_background.store(true, std::memory_order_release);
+
+ if (_update_ttl_thread.joinable()) {
+ _update_ttl_thread.join();
+ }
+
+ if (_expiration_check_thread.joinable()) {
+ _expiration_check_thread.join();
+ }
+}
+
+void BlockFileCacheTtlMgr::register_tablet_id(int64_t tablet_id) {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ _tablet_id_set.insert(tablet_id);
+}
+
+FileBlocks BlockFileCacheTtlMgr::get_file_blocks_from_tablet_id(int64_t tablet_id) {
+ FileBlocks result;
+
+ // Use meta store to get all blocks for this tablet
+ auto iterator = _meta_store->range_get(tablet_id);
+ if (!iterator) {
+ LOG(WARNING) << "Failed to get iterator for tablet_id: " << tablet_id;
+ return result;
+ }
+
+ while (iterator->valid()) {
+ BlockMetaKey key = iterator->key();
+ BlockMeta meta = iterator->value();
+
+ // Get all blocks for this hash using get_blocks_by_key
+ try {
+ auto blocks_map = _mgr->get_blocks_by_key(key.hash);
+ for (const auto& [offset, block] : blocks_map) {
+ // Only add blocks that match our specific offset
+ if (offset == key.offset) {
+ result.push_back(block);
+ break;
+ }
+ }
+ } catch (const std::exception& e) {
+ LOG(WARNING) << "Failed to get file blocks for tablet_id: " << tablet_id
+ << ", hash: " << key.hash.to_string() << ", error: " << e.what();
+ }
+
+ iterator->next();
+ }
+
+ return result;
+}
+
+void BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map() {
+ Thread::set_self_name("ttl_mgr_update");
+
+ while (!_stop_background.load(std::memory_order_acquire)) {
+ try {
+ std::unordered_set<int64_t> tablet_ids_to_process;
+
+ // Copy tablet IDs to process
+ {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ tablet_ids_to_process = _tablet_id_set;
+ }
+
+ for (int64_t tablet_id : tablet_ids_to_process) {
+ // TODO(zhengyu): Implement actual CloudMetaMgr or CloudTabletMgr integration
+ // For now, we'll use placeholder values
+ uint64_t create_time = 0;
+ uint64_t ttl = 0;
+
+ // Simulate getting tablet metadata
+ // This should be replaced with actual CloudMetaMgr::get_tablet_meta call
Review Comment:
return_error_if_miss , skip_cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494308929
##########
be/src/io/cache/block_file_cache_ttl_mgr.h:
##########
@@ -0,0 +1,69 @@
+
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+#include <map>
+#include <mutex>
+#include <set>
+#include <thread>
+#include <unordered_set>
+
+#include "io/cache/file_cache_common.h"
+
+namespace doris::io {
+
+class BlockFileCache;
+class CacheBlockMetaStore;
+
+struct TtlInfo {
+ uint64_t ttl;
+ uint64_t create_time;
+};
+
+class BlockFileCacheTtlMgr {
+public:
+ BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store);
+ ~BlockFileCacheTtlMgr();
+
+ void register_tablet_id(int64_t tablet_id);
+
+ // Background thread functions
+ void run_backgroud_update_ttl_info_map();
+ void run_backgroud_expiration_check();
+
+private:
+ FileBlocks get_file_blocks_from_tablet_id(int64_t tablet_id);
+
+private:
+ std::unordered_set<int64_t> _tablet_id_set;
Review Comment:
只增不减
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer closed pull request #57647: [refactor](filecache) ttl management refactoring
URL: https://github.com/apache/doris/pull/57647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494320801
##########
be/src/io/cache/block_file_cache_ttl_mgr.h:
##########
@@ -0,0 +1,69 @@
+
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+#include <map>
+#include <mutex>
+#include <set>
+#include <thread>
+#include <unordered_set>
+
+#include "io/cache/file_cache_common.h"
+
+namespace doris::io {
+
+class BlockFileCache;
+class CacheBlockMetaStore;
+
+struct TtlInfo {
+ uint64_t ttl;
+ uint64_t create_time;
Review Comment:
tablet_ctime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2492835909
##########
be/src/io/cache/block_file_cache_ttl_mgr.cpp:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/block_file_cache_ttl_mgr.h"
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <thread>
+#include <unordered_set>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/cache_block_meta_store.h"
+#include "io/cache/file_block.h"
+#include "util/time.h"
+
+namespace doris::io {
+
+BlockFileCacheTtlMgr::BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store)
+ : _mgr(mgr), _meta_store(meta_store), _stop_background(false) {
+ // Start background threads
+ _update_ttl_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map, this);
+ _expiration_check_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_expiration_check, this);
+}
+
+BlockFileCacheTtlMgr::~BlockFileCacheTtlMgr() {
+ _stop_background.store(true, std::memory_order_release);
+
+ if (_update_ttl_thread.joinable()) {
+ _update_ttl_thread.join();
+ }
+
+ if (_expiration_check_thread.joinable()) {
+ _expiration_check_thread.join();
+ }
+}
+
+void BlockFileCacheTtlMgr::register_tablet_id(int64_t tablet_id) {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ _tablet_id_set.insert(tablet_id);
+}
+
+FileBlocks BlockFileCacheTtlMgr::get_file_blocks_from_tablet_id(int64_t tablet_id) {
+ FileBlocks result;
+
+ // Use meta store to get all blocks for this tablet
+ auto iterator = _meta_store->range_get(tablet_id);
+ if (!iterator) {
+ LOG(WARNING) << "Failed to get iterator for tablet_id: " << tablet_id;
+ return result;
+ }
+
+ while (iterator->valid()) {
+ BlockMetaKey key = iterator->key();
+ BlockMeta meta = iterator->value();
+
+ // Get all blocks for this hash using get_blocks_by_key
+ try {
+ auto blocks_map = _mgr->get_blocks_by_key(key.hash);
+ for (const auto& [offset, block] : blocks_map) {
+ // Only add blocks that match our specific offset
+ if (offset == key.offset) {
+ result.push_back(block);
+ break;
+ }
+ }
+ } catch (const std::exception& e) {
+ LOG(WARNING) << "Failed to get file blocks for tablet_id: " << tablet_id
+ << ", hash: " << key.hash.to_string() << ", error: " << e.what();
+ }
+
+ iterator->next();
+ }
+
+ return result;
+}
+
+void BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map() {
+ Thread::set_self_name("ttl_mgr_update");
+
+ while (!_stop_background.load(std::memory_order_acquire)) {
+ try {
+ std::unordered_set<int64_t> tablet_ids_to_process;
+
+ // Copy tablet IDs to process
+ {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ tablet_ids_to_process = _tablet_id_set;
+ }
+
+ for (int64_t tablet_id : tablet_ids_to_process) {
+ // TODO(zhengyu): Implement actual CloudMetaMgr or CloudTabletMgr integration
+ // For now, we'll use placeholder values
+ uint64_t create_time = 0;
+ uint64_t ttl = 0;
+
+ // Simulate getting tablet metadata
+ // This should be replaced with actual CloudMetaMgr::get_tablet_meta call
+ bool has_ttl = (tablet_id % 10 == 0); // Example condition
+ if (has_ttl) {
+ create_time = UnixSeconds(); // Current time as create time
+ ttl = 3600; // 1 hour TTL
+ }
+
+ // Update TTL info map
+ {
+ std::lock_guard<std::mutex> lock(_ttl_info_mutex);
+ if (ttl > 0) {
+ _ttl_info_map[tablet_id] = TtlInfo {ttl, create_time};
+
+ // If TTL changed from 0 to non-zero, convert blocks to TTL type
+ auto old_info_it = _ttl_info_map.find(tablet_id);
+ if (old_info_it == _ttl_info_map.end() || old_info_it->second.ttl == 0) {
+ FileBlocks blocks = get_file_blocks_from_tablet_id(tablet_id);
+ for (auto& block : blocks) {
+ if (block->cache_type() != FileCacheType::TTL) {
+ block->change_cache_type(FileCacheType::TTL);
+ }
+ }
+ }
+ } else {
+ // Remove from TTL map if TTL is 0
+ _ttl_info_map.erase(tablet_id);
+ }
+ }
+ }
+
+ // Sleep for configured interval (use existing TTL GC interval)
+ std::this_thread::sleep_for(
+ std::chrono::milliseconds(config::file_cache_background_ttl_gc_interval_ms));
Review Comment:
改成每 3 分钟一检查
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494372289
##########
be/src/io/cache/block_file_cache.cpp:
##########
@@ -717,6 +636,7 @@ FileBlocks BlockFileCache::split_range_into_cells(const UInt128Wrapper& hash,
cell->update_atime();
}
}
+ _ttl_mgr->register_tablet_id(context.tablet_id);
Review Comment:
这里频率需要控制，可能有锁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #57647:
URL: https://github.com/apache/doris/pull/57647#issuecomment-3516755627
updated version (rebased) in #57922
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494323992
##########
be/src/io/cache/block_file_cache_ttl_mgr.h:
##########
@@ -0,0 +1,69 @@
+
+
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <atomic>
+#include <map>
+#include <mutex>
+#include <set>
+#include <thread>
+#include <unordered_set>
+
+#include "io/cache/file_cache_common.h"
+
+namespace doris::io {
+
+class BlockFileCache;
+class CacheBlockMetaStore;
+
+struct TtlInfo {
+ uint64_t ttl;
+ uint64_t create_time;
+};
+
+class BlockFileCacheTtlMgr {
+public:
+ BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store);
+ ~BlockFileCacheTtlMgr();
+
+ void register_tablet_id(int64_t tablet_id);
+
+ // Background thread functions
+ void run_backgroud_update_ttl_info_map();
Review Comment:
comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](filecache) ttl management refactoring [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on code in PR #57647:
URL: https://github.com/apache/doris/pull/57647#discussion_r2494419445
##########
be/src/io/cache/block_file_cache_ttl_mgr.cpp:
##########
@@ -0,0 +1,200 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "io/cache/block_file_cache_ttl_mgr.h"
+
+#include <atomic>
+#include <chrono>
+#include <memory>
+#include <mutex>
+#include <thread>
+#include <unordered_set>
+
+#include "common/config.h"
+#include "common/logging.h"
+#include "io/cache/block_file_cache.h"
+#include "io/cache/cache_block_meta_store.h"
+#include "io/cache/file_block.h"
+#include "util/time.h"
+
+namespace doris::io {
+
+BlockFileCacheTtlMgr::BlockFileCacheTtlMgr(BlockFileCache* mgr, CacheBlockMetaStore* meta_store)
+ : _mgr(mgr), _meta_store(meta_store), _stop_background(false) {
+ // Start background threads
+ _update_ttl_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map, this);
+ _expiration_check_thread =
+ std::thread(&BlockFileCacheTtlMgr::run_backgroud_expiration_check, this);
+}
+
+BlockFileCacheTtlMgr::~BlockFileCacheTtlMgr() {
+ _stop_background.store(true, std::memory_order_release);
+
+ if (_update_ttl_thread.joinable()) {
+ _update_ttl_thread.join();
+ }
+
+ if (_expiration_check_thread.joinable()) {
+ _expiration_check_thread.join();
+ }
+}
+
+void BlockFileCacheTtlMgr::register_tablet_id(int64_t tablet_id) {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ _tablet_id_set.insert(tablet_id);
+}
+
+FileBlocks BlockFileCacheTtlMgr::get_file_blocks_from_tablet_id(int64_t tablet_id) {
+ FileBlocks result;
+
+ // Use meta store to get all blocks for this tablet
+ auto iterator = _meta_store->range_get(tablet_id);
+ if (!iterator) {
+ LOG(WARNING) << "Failed to get iterator for tablet_id: " << tablet_id;
+ return result;
+ }
+
+ while (iterator->valid()) {
+ BlockMetaKey key = iterator->key();
+ BlockMeta meta = iterator->value();
+
+ // Get all blocks for this hash using get_blocks_by_key
+ try {
+ auto blocks_map = _mgr->get_blocks_by_key(key.hash);
+ for (const auto& [offset, block] : blocks_map) {
+ // Only add blocks that match our specific offset
+ if (offset == key.offset) {
+ result.push_back(block);
+ break;
+ }
+ }
+ } catch (const std::exception& e) {
+ LOG(WARNING) << "Failed to get file blocks for tablet_id: " << tablet_id
+ << ", hash: " << key.hash.to_string() << ", error: " << e.what();
+ }
+
+ iterator->next();
+ }
+
+ return result;
+}
+
+void BlockFileCacheTtlMgr::run_backgroud_update_ttl_info_map() {
+ Thread::set_self_name("ttl_mgr_update");
+
+ while (!_stop_background.load(std::memory_order_acquire)) {
+ try {
+ std::unordered_set<int64_t> tablet_ids_to_process;
+
+ // Copy tablet IDs to process
+ {
+ std::lock_guard<std::mutex> lock(_tablet_id_mutex);
+ tablet_ids_to_process = _tablet_id_set;
+ }
+
+ for (int64_t tablet_id : tablet_ids_to_process) {
+ // TODO(zhengyu): Implement actual CloudMetaMgr or CloudTabletMgr integration
+ // For now, we'll use placeholder values
+ uint64_t create_time = 0;
+ uint64_t ttl = 0;
+
+ // Simulate getting tablet metadata
+ // This should be replaced with actual CloudMetaMgr::get_tablet_meta call
Review Comment:
meta mgr 有个 CloudTabletMgr::get_tablet(int64_t tablet_id, bool warmup_data,
bool sync_delete_bitmap,
SyncRowsetStats* sync_stats,
bool local_only) { // use local only to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org