You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2021/07/13 03:15:16 UTC
[GitHub] [incubator-pegasus] levy5307 opened a new pull request #780: feat: start compaction operation in CompactionFilter::Filter
levy5307 opened a new pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780
### What problem does this PR solve? <!--add issue link with summary if exists-->
start compaction operation in CompactionFilter::Filter
RFC: https://github.com/apache/incubator-pegasus/blob/master/rfcs/2021-05-27-user-specified-compaction.md
### What is changed and how does it work?
### Checklist <!--REMOVE the items that are not applicable-->
##### Tests <!-- At least one of them must be included. -->
- Manual test (add detailed scripts or steps below)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669331346
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
It will extract field from key in the compaction operation. That will cause coredump if key.size() < 2,
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
It will extract field from key in the compaction operation. That will cause coredump if key.size() < 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669331346
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
It will extract fields from key in the compaction operation. That will cause coredump if key.size() < 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669331346
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
In the compaction operation, It will extract field from key. If key.size() < 2, it will cause coredump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669331346
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
It will extract fields from key in the compaction operation. That will cause coredump if key.size() < 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 merged pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 merged pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 merged pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 merged pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 merged pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 merged pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] levy5307 commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
levy5307 commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669331346
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
It will extract fields from key in the compaction operation. That will cause coredump if key.size() < 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] Shuo-Jia commented on a change in pull request #780: feat: start compaction operation in CompactionFilter::Filter
Posted by GitBox <gi...@apache.org>.
Shuo-Jia commented on a change in pull request #780:
URL: https://github.com/apache/incubator-pegasus/pull/780#discussion_r669315408
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
Review comment:
if user's key < 2 actually?
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -60,6 +62,23 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
return false;
}
+ // ignore empty write
+ if (key.size() < 2) {
+ return false;
+ }
+
+ // user specified compaction operations
+ if (!_user_specified_operations.empty()) {
+ std::string hash_key, sort_key;
+ pegasus_restore_key(dsn::blob(key.data(), 0, key.size()), hash_key, sort_key);
+ for (const auto &op : _user_specified_operations) {
+ if (op->filter(hash_key, sort_key, existing_value, new_value, value_changed)) {
+ // return true if this data need to be deleted
+ return true;
+ }
+ }
+ }
+
uint32_t expire_ts =
Review comment:
suggest follow for undestanding the compaction action:
```c++
return user_specified_operations_filter || records_ttl_expired
function user_specified_operations_filter() {
return op->filter(hash_key, sort_key, existing_value, new_value, value_changed)
}
function records_ttl_expired() {
return check_if_ts_expired(utils::epoch_now(), expire_ts) || check_if_stale_split_data(key);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org