You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@resilientdb.apache.org by "harish876 (via GitHub)" <gi...@apache.org> on 2025/02/12 11:42:41 UTC
[PR] MemLens Database Specific Changes [incubator-resilientdb]
harish876 opened a new pull request, #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170
## Summary:
- Integrated a Stats Class with LevelDB to export internal statistics, including LRU cache hit-and-miss metrics for the Memory Engine.
- Added an LRU Cache controlled via a switch to track and report cache hit-and-miss ratios as part of the storage engine metrics on the front end.
- Implemented the generation of `compile_commands.json` to enable autocompletion and IntelliSense during development.
- Introduced debug flags during build steps for better profiling and performance tracking.
- Fixed a bug related to passing the LevelDB configuration to the KV store.
- Created a new endpoint `/transaction_data` to export specific statistics to the frontend.
##Todo:
Bootstrap script like ./INSTALL.sh to install and setup all monitoring tools
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954300110
##########
chain/storage/leveldb.cpp:
##########
@@ -50,6 +52,14 @@ ResLevelDB::ResLevelDB(std::optional<LevelDBInfo> config) {
path = (*config).path();
}
}
+ if ((*config).enable_block_cache()) {
+ LRUCache<std::string, std::string>* block_cache =
+ new LRUCache<std::string, std::string>(1000);
+ block_cache_ =
Review Comment:
```suggestion
block_cache_ =
std::make_unique<LRUCache<std::string, std::string>>(1000);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen merged PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2707242373
yes @cjcchen . it was a mistake. corrected it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2726455293
> Hi @cjcchen , verified the build and unit tests again. I removed our profiling command, which needs to be installed. I will convert our setup to a sidecar. Please find my deployment results. I also have enabled LevelDB here, please do let me know if that is fine by you
>
> <img alt="Screenshot 2025-03-15 at 1 43 49 AM" width="2000" src="https://private-user-images.githubusercontent.com/56532575/423058073-d3ad965f-d1e3-4393-96a6-d9384baeac71.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDIwMzc2MTIsIm5iZiI6MTc0MjAzNzMxMiwicGF0aCI6Ii81NjUzMjU3NS80MjMwNTgwNzMtZDNhZDk2NWYtZDFlMy00MzkzLTk2YTYtZDkzODRiYWVhYzcxLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzE1VDExMTUxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAyZGI3ZTYyNWZmNmI0MTQwMmY1ZjU2OGQzNjY0Nzk3NzQ5M2Q1Yzc3NDJmMzk1ZWJlZDlmMGNlYWViMWJhOWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.lPEgM4E6vxiyC1D4lkEOhZmwK4lsKlUaiFFW1AGmKew"> <img alt="Screenshot 2025-03-15 at 1 52 22 AM" width="2000" src="https://private-user-images.githubusercontent.com/56532575/423058089-83fe10b5-
23f9-46a0-a58d-e45c1fc8f99b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDIwMzc2MTIsIm5iZiI6MTc0MjAzNzMxMiwicGF0aCI6Ii81NjUzMjU3NS80MjMwNTgwODktODNmZTEwYjUtMjNmOS00NmEwLWE1OGQtZTQ1YzFmYzhmOTliLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAzMTUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMzE1VDExMTUxMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTRlYTM1Nzk1ZTlkNWUxMzk5ODA3MzFlYWE3Mjk1NGYzYjljYjc2NWY1MjU0NjA5YzgxMWUxNGYzMjc5OTg2MzkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.gSaYJELsoYCriexLj3a5U9lFMTu7wZMcy9HjMXwrNAk">
"common/lru/lru_cache.cpp:21:1: note: 'std::string' is defined in header '<string>'; did you forget to '#include <string>'?
"
it reports this. I think you just need to include <string> to the header.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2705856946
Hi @cjcchen tried to resolve the comments. please do review once and let me know if further changes need to be made. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2723881551
Hi @cjcchen. I have removed the debug flags. Our profiler only works when the build has debug information at the top level. I am unsure if profiling would work without the -pg flag. We do get kernel level metrics without the debug flag but userspace metrics are not available without that.
Please do let us know what could be done here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2675827957
> Hi @cjcchen , I have made the necessary changes pointed out by you. Can you please review them?
Very nice job.
Left a few comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2656289159
Hi @harish876,
Thanks for the contribution and it is a nice accomplishment.
I left some comments. Please take a look.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954303483
##########
common/lru/lru_cache.cpp:
##########
@@ -0,0 +1,101 @@
+#include "lru_cache.h"
+
+#include <algorithm>
+
+namespace resdb {
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::LRUCache(int capacity) {
+ m_ = capacity;
+ cache_hits_ = 0;
+ cache_misses_ = 0;
+}
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::~LRUCache() {
+ um_.clear();
+ dq_.clear();
+}
+
+template <typename KeyType, typename ValueType>
+ValueType LRUCache<KeyType, ValueType>::Get(KeyType key) {
+ if (!um_.count(key)) {
+ cache_misses_++;
+ return ValueType();
+ }
+
+ auto it = std::find(dq_.begin(), dq_.end(), key);
+ dq_.erase(it);
Review Comment:
```suggestion
if ( it != dq_.end()){
dq_.erase(it);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954311574
##########
common/lru/lru_cache.cpp:
##########
@@ -0,0 +1,101 @@
+#include "lru_cache.h"
Review Comment:
Please add some UTs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966285377
##########
gmon.out:
##########
Review Comment:
I think this file won't be committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954294029
##########
chain/storage/leveldb.cpp:
##########
@@ -50,6 +52,14 @@ ResLevelDB::ResLevelDB(std::optional<LevelDBInfo> config) {
path = (*config).path();
}
}
+ if ((*config).enable_block_cache()) {
+ LRUCache<std::string, std::string>* block_cache =
Review Comment:
When to destroy this pointer?
It looks like there is a memory leak.
You can use unique_pointer instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1985905988
##########
service/tools/kv/api_tools/BUILD:
##########
@@ -27,6 +27,8 @@ cc_binary(
"//interface/kv:kv_client",
"//platform/config:resdb_config_utils",
],
+ copts = ["-pg"],
+ linkopts = ["-pg"]
Review Comment:
do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966294250
##########
common/lru/lru_cache.cpp:
##########
@@ -0,0 +1,97 @@
+#include "lru_cache.h"
+
+namespace resdb {
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::LRUCache(int capacity) {
+ capacity_ = capacity;
+ cache_hits_ = 0;
+ cache_misses_ = 0;
+}
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::~LRUCache() {
+ lookup_.clear();
+ key_list_.clear();
+ rlookup_.clear();
+}
+
+template <typename KeyType, typename ValueType>
+ValueType LRUCache<KeyType, ValueType>::Get(KeyType key) {
+ if (lookup_.find(key) == lookup_.end()) {
+ cache_misses_++;
+ return ValueType();
+ }
+
+ // Move accessed key to front of key list. This marks the key as used
+ key_list_.splice(key_list_.begin(), key_list_, rlookup_[key]);
+
+ cache_hits_++;
+ return lookup_[key];
+}
+
+template <typename KeyType, typename ValueType>
+void LRUCache<KeyType, ValueType>::Put(KeyType key, ValueType value) {
+ if (lookup_.find(key) == lookup_.end()) {
+ if (key_list_.size() == capacity_) {
+ KeyType lru_key = key_list_.back();
+ key_list_.pop_back();
+ lookup_.erase(lru_key);
+ rlookup_.erase(lru_key);
+ }
+ key_list_.push_front(key);
+ rlookup_[key] = key_list_.begin();
+ } else {
+ lookup_[key] = value;
Review Comment:
could we remove this? you will set the lookup_ below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966286945
##########
common/lru/lru_cache.h:
##########
Review Comment:
Let's add the ASF header here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2726479117
Done @cjcchen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2724267562
> Hi @cjcchen. I have removed the debug flags. Our profiler only works when the build has debug information at the top level. I am unsure if profiling would work without the -pg flag. We do get kernel level metrics without the debug flag but userspace metrics are not available without that.
>
> Please do let us know what could be done here?
@harish876 did it build successfully on your end?
https://github.com/apache/incubator-resilientdb/actions/runs/13852115218/job/38763569382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2706770502
> Hi @cjcchen tried to resolve the comments. please do review once and let me know if further changes need to be made. thanks
Very good job.
One question here, what is the motivation to remove the O3 option?
Do we need to always open this feature?
If this is optional to just for the profile tracing, we can add an introduction on how to enable the feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966298441
##########
platform/statistic/stats.cpp:
##########
@@ -96,6 +96,30 @@ Stats::~Stats() {
}
}
+long getRSS() {
+ long rss = 0;
+ FILE* fp = NULL;
+ if ((fp = fopen("/proc/self/statm", "r")) == NULL) {
+ return 0;
+ }
+
+ unsigned long size, resident, share, text, lib, data, dt;
Review Comment:
let's use int64_t and uint64_t other than long and unsigned long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2724019310
> Hi @cjcchen. I have removed the debug flags. Our profiler only works when the build has debug information at the top level. I am unsure if profiling would work without the -pg flag. We do get kernel level metrics without the debug flag but userspace metrics are not available without that.
>
> Please do let us know what could be done here?
Very cool!
Merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2726362541
Hi @cjcchen , verified the build and unit tests again. I removed our profiling command, which needs to be installed. I will convert our setup to a sidecar. Please find my deployment results
<img width="2048" alt="Screenshot 2025-03-15 at 1 43 49 AM" src="https://github.com/user-attachments/assets/d3ad965f-d1e3-4393-96a6-d9384baeac71" />
<img width="2048" alt="Screenshot 2025-03-15 at 1 52 22 AM" src="https://github.com/user-attachments/assets/83fe10b5-23f9-46a0-a58d-e45c1fc8f99b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2675382113
Hi @cjcchen , I have made the necessary changes pointed out by you. Can you please review them?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966299227
##########
platform/statistic/stats.cpp:
##########
@@ -182,6 +241,15 @@ void Stats::SetPrimaryId(int primary_id) {
transaction_summary_.primary_id = primary_id;
}
+void Stats::SetStorageEngineMetrics(double ext_cache_hit_ratio,
Review Comment:
format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966297553
##########
platform/statistic/stats.cpp:
##########
@@ -96,6 +96,30 @@ Stats::~Stats() {
}
}
+long getRSS() {
Review Comment:
```suggestion
long GetRSS() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966283934
##########
chain/storage/leveldb.cpp:
##########
@@ -94,7 +108,16 @@ int ResLevelDB::SetValue(const std::string& key, const std::string& value) {
std::string ResLevelDB::GetValue(const std::string& key) {
std::string value = "";
+ std::string cached_result = "";
Review Comment:
something wrong here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2657390547
Hi @cjcchen
Thanks for the comments. I will go through them and resolve them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954307163
##########
common/lru/lru_cache.h:
##########
@@ -0,0 +1,46 @@
+#include <deque>
+#include <unordered_map>
+
+using namespace std;
+
+namespace resdb {
+
+template <typename KeyType, typename ValueType>
+class LRUCache {
+ public:
+ // Constructor to initialize the cache with a given capacity
+ LRUCache(int capacity);
+
+ // Destructor
+ ~LRUCache();
+
+ // Get the value of the key if present in the cache
+ ValueType Get(KeyType key);
+
+ // Insert or update the key-value pair
+ void Put(KeyType key, ValueType value);
+
+ // Method to change the cache capacity
+ void SetCapacity(int new_capacity);
+
+ // Method to flush the cache (clear all entries)
+ void Flush();
+
+ // Method to get the cache hit count
+ int GetCacheHits() const;
+
+ // Method to get the cache miss count
+ int GetCacheMisses() const;
+
+ // Method to get the cache Hit Ratio
+ double GetCacheHitRatio() const;
+
+ private:
+ int m_; // Cache capacity
+ int cache_hits_; // Cache hits count
+ int cache_misses_; // Cache misses count
+
+ deque<KeyType> dq_; // To maintain most and least recently used items
Review Comment:
why not use link-list?
https://en.cppreference.com/w/cpp/container/list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2724293625
Yes @cjcchen , it did build with warnings albeit, but all TC's were a go. I will go through the failed checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954302894
##########
common/lru/lru_cache.cpp:
##########
@@ -0,0 +1,101 @@
+#include "lru_cache.h"
+
+#include <algorithm>
+
+namespace resdb {
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::LRUCache(int capacity) {
+ m_ = capacity;
+ cache_hits_ = 0;
+ cache_misses_ = 0;
+}
+
+template <typename KeyType, typename ValueType>
+LRUCache<KeyType, ValueType>::~LRUCache() {
+ um_.clear();
+ dq_.clear();
+}
+
+template <typename KeyType, typename ValueType>
+ValueType LRUCache<KeyType, ValueType>::Get(KeyType key) {
+ if (!um_.count(key)) {
+ cache_misses_++;
+ return ValueType();
+ }
+
+ auto it = std::find(dq_.begin(), dq_.end(), key);
Review Comment:
performance issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1985270857
##########
.bazelrc:
##########
@@ -1,4 +1,4 @@
-build --cxxopt='-std=c++17' --copt=-O3 --jobs=40
Review Comment:
???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "harish876 (via GitHub)" <gi...@apache.org>.
harish876 commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2689339321
Thanks @cjcchen , will resolve the additional comments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1985905826
##########
service/kv/BUILD:
##########
@@ -25,17 +25,19 @@ cc_binary(
name = "kv_service",
srcs = ["kv_service.cpp"],
copts = select({
- "//chain/storage/setting:enable_leveldb_setting": ["-DENABLE_LEVELDB"],
- "//conditions:default": [],
+ "//chain/storage/setting:enable_leveldb_setting": ["-DENABLE_LEVELDB","-g","-ggdb"],
+ "//conditions:default": ["-pg", "-g", "-ggdb"],
Review Comment:
How about this -pg -ggdb?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#issuecomment-2707837447
> yes @cjcchen . it was a mistake. corrected it now
Great.
How about the "pg" in the BUILD files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966303778
##########
platform/statistic/stats.h:
##########
@@ -60,6 +61,14 @@ struct VisualData {
std::vector<std::chrono::system_clock::time_point>
commit_message_count_times_list;
std::chrono::system_clock::time_point execution_time;
+
+ // Storage Engine Stats
+ double ext_cache_hit_ratio;
Review Comment:
some naming issues. using "_" as the subfix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1966286328
##########
chain/storage/leveldb.cpp:
##########
Review Comment:
let's add some UTs for the leveldb.cpp that relevant to your code :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954289246
##########
chain/storage/BUILD:
##########
@@ -47,11 +47,14 @@ cc_library(
name = "leveldb",
srcs = ["leveldb.cpp"],
hdrs = ["leveldb.h"],
+ linkopts = ["-pg","-g","-ggdb"], # Enable profiling during linking
Review Comment:
Only open this debug opts if needed.
Most of time, we dont to debug the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MemLens Database Specific Changes [incubator-resilientdb]
Posted by "cjcchen (via GitHub)" <gi...@apache.org>.
cjcchen commented on code in PR #170:
URL: https://github.com/apache/incubator-resilientdb/pull/170#discussion_r1954286407
##########
.bazelrc:
##########
@@ -1,4 +1,4 @@
-build --cxxopt='-std=c++17' --copt=-O3 --jobs=40
Review Comment:
please do not change the compile settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@resilientdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org