You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2020/10/12 09:29:01 UTC

[GitHub] [incubator-pegasus] Smityz opened a new pull request #615: feat(hotkey): add an interface of hotkey capture

Smityz opened a new pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615


   hotkey capturing is executed in `capacity_unit_calculator`.
   Both `capture_raw_key` and `capture_hash_key` have two params: `key` `weight`
   `key`: raw_key or hash_key
   `weight`: In some scenarios, some keys will be counted several times, we can capture these keys just one time.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503645992



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -110,9 +123,11 @@ void capacity_unit_calculator::add_multi_get_cu(int32_t status,
 {
     int64_t data_size = 0;
     int64_t multi_get_bytes = 0;
+    int64_t key_count = 0;

Review comment:
       `uint64_t key_count = kvs.size();`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] zhangyifan27 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
zhangyifan27 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503640382



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -144,25 +161,28 @@ void capacity_unit_calculator::add_scan_cu(int32_t status,
     int64_t data_size = 0;
     for (const auto &kv : kvs) {
         data_size += kv.key.size() + kv.value.size();
+        _read_hotkey_collector->capture_raw_key(kv.key, 1);
     }
     add_read_cu(data_size);
     _pfc_scan_bytes->add(data_size);
 }
 
-void capacity_unit_calculator::add_sortkey_count_cu(int32_t status)
+void capacity_unit_calculator::add_sortkey_count_cu(int32_t status, const dsn::blob &hash_key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_hash_key(hash_key, 1);
 }
 
-void capacity_unit_calculator::add_ttl_cu(int32_t status)
+void capacity_unit_calculator::add_ttl_cu(int32_t status, const dsn::blob &key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_raw_key(key, 1);

Review comment:
       If these two functions are always called at the same time, you can put them into a new function `collect_hotkey_and_cu`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503629470



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -144,25 +161,28 @@ void capacity_unit_calculator::add_scan_cu(int32_t status,
     int64_t data_size = 0;
     for (const auto &kv : kvs) {
         data_size += kv.key.size() + kv.value.size();
+        _read_hotkey_collector->capture_raw_key(kv.key, 1);
     }
     add_read_cu(data_size);
     _pfc_scan_bytes->add(data_size);
 }
 
-void capacity_unit_calculator::add_sortkey_count_cu(int32_t status)
+void capacity_unit_calculator::add_sortkey_count_cu(int32_t status, const dsn::blob &hash_key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_hash_key(hash_key, 1);
 }
 
-void capacity_unit_calculator::add_ttl_cu(int32_t status)
+void capacity_unit_calculator::add_ttl_cu(int32_t status, const dsn::blob &key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_raw_key(key, 1);

Review comment:
       I mean you should wrap these two lines of code into a new function, but not merge `_read_hotkey_collector->capture_raw_key(key, 1);` into `add_read_cu(1)`.
   Because these two lines are always called in the same time.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503629470



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -144,25 +161,28 @@ void capacity_unit_calculator::add_scan_cu(int32_t status,
     int64_t data_size = 0;
     for (const auto &kv : kvs) {
         data_size += kv.key.size() + kv.value.size();
+        _read_hotkey_collector->capture_raw_key(kv.key, 1);
     }
     add_read_cu(data_size);
     _pfc_scan_bytes->add(data_size);
 }
 
-void capacity_unit_calculator::add_sortkey_count_cu(int32_t status)
+void capacity_unit_calculator::add_sortkey_count_cu(int32_t status, const dsn::blob &hash_key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_hash_key(hash_key, 1);
 }
 
-void capacity_unit_calculator::add_ttl_cu(int32_t status)
+void capacity_unit_calculator::add_ttl_cu(int32_t status, const dsn::blob &key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_raw_key(key, 1);

Review comment:
       I mean you should wrap these two lines of code into a new function, but not merge `_read_hotkey_collector->capture_raw_key(key, 1)` into `add_read_cu(1)`.
   Because these two lines are always called in the same time.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503646434



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -256,9 +284,11 @@ void capacity_unit_calculator::add_check_and_mutate_cu(
 {
     int64_t data_size = 0;
     int64_t check_and_mutate_bytes = 0;
+    int64_t key_count = 0;

Review comment:
       `uint64_t key_count = mutate_list.size();`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503645992



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -110,9 +123,11 @@ void capacity_unit_calculator::add_multi_get_cu(int32_t status,
 {
     int64_t data_size = 0;
     int64_t multi_get_bytes = 0;
+    int64_t key_count = 0;

Review comment:
       int64_t key_count = kvs.size();




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503214929



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -110,9 +123,11 @@ void capacity_unit_calculator::add_multi_get_cu(int32_t status,
 {
     int64_t data_size = 0;
     int64_t multi_get_bytes = 0;
+    int64_t key_count = 0;

Review comment:
       uint64_t




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] Smityz commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
Smityz commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503627225



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -144,25 +161,28 @@ void capacity_unit_calculator::add_scan_cu(int32_t status,
     int64_t data_size = 0;
     for (const auto &kv : kvs) {
         data_size += kv.key.size() + kv.value.size();
+        _read_hotkey_collector->capture_raw_key(kv.key, 1);
     }
     add_read_cu(data_size);
     _pfc_scan_bytes->add(data_size);
 }
 
-void capacity_unit_calculator::add_sortkey_count_cu(int32_t status)
+void capacity_unit_calculator::add_sortkey_count_cu(int32_t status, const dsn::blob &hash_key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_hash_key(hash_key, 1);
 }
 
-void capacity_unit_calculator::add_ttl_cu(int32_t status)
+void capacity_unit_calculator::add_ttl_cu(int32_t status, const dsn::blob &key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_raw_key(key, 1);

Review comment:
       `add_read_cu(1);` and `_read_hotkey_collector->capture_raw_key(key, 1);` have different semantics, they are not suitable for merging
   @zhangyifan27 
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 merged pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 merged pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org


[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #615: feat(hotkey): add an interface of hotkey capture

Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #615:
URL: https://github.com/apache/incubator-pegasus/pull/615#discussion_r503215436



##########
File path: src/server/capacity_unit_calculator.cpp
##########
@@ -144,25 +161,28 @@ void capacity_unit_calculator::add_scan_cu(int32_t status,
     int64_t data_size = 0;
     for (const auto &kv : kvs) {
         data_size += kv.key.size() + kv.value.size();
+        _read_hotkey_collector->capture_raw_key(kv.key, 1);
     }
     add_read_cu(data_size);
     _pfc_scan_bytes->add(data_size);
 }
 
-void capacity_unit_calculator::add_sortkey_count_cu(int32_t status)
+void capacity_unit_calculator::add_sortkey_count_cu(int32_t status, const dsn::blob &hash_key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_hash_key(hash_key, 1);
 }
 
-void capacity_unit_calculator::add_ttl_cu(int32_t status)
+void capacity_unit_calculator::add_ttl_cu(int32_t status, const dsn::blob &key)
 {
     if (status != rocksdb::Status::kOk && status != rocksdb::Status::kNotFound) {
         return;
     }
     add_read_cu(1);
+    _read_hotkey_collector->capture_raw_key(key, 1);

Review comment:
       refactor to move L184~L185 to a function




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org