You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pegasus.apache.org by GitBox <gi...@apache.org> on 2021/03/12 08:21:10 UTC
[GitHub] [incubator-pegasus] hycdong opened a new pull request #698: feat(split): gc useless data after partition split
hycdong opened a new pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698
### What problem does this PR solve? <!--add issue link with summary if exists-->
### What is changed and how it works?
### Check List <!--REMOVE the items that are not applicable-->
Tests <!-- At least one of them must be included. -->
- Unit test
- Integration test
- Manual test (add detailed scripts or steps below)
- No code
Code changes
- Has exported function/method change
- Has exported variable/fields change
- Has interface methods change
- Has persistent data change
Side effects
- Possible performance regression
- Increased code complexity
- Breaking backward compatibility
Related changes
- Need to cherry-pick to the release branch
- Need to update the documentation
- Need to be included in the release note
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] acelyc111 merged pull request #698: feat(split): gc useless data after partition split
Posted by GitBox <gi...@apache.org>.
acelyc111 merged pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] neverchanje commented on a change in pull request #698: feat(split): gc useless data after partition split
Posted by GitBox <gi...@apache.org>.
neverchanje commented on a change in pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698#discussion_r598510187
##########
File path: src/server/key_ttl_compaction_filter.h
##########
@@ -58,16 +69,28 @@ class KeyWithTTLCompactionFilter : public rocksdb::CompactionFilter
*value_changed = true;
return false;
}
- return check_if_ts_expired(utils::epoch_now(), expire_ts);
+ return check_if_ts_expired(utils::epoch_now(), expire_ts) || check_if_stale_split_data(key);
}
const char *Name() const override { return "KeyWithTTLCompactionFilter"; }
+ bool check_if_stale_split_data(const rocksdb::Slice &key) const
Review comment:
```suggestion
// Check if the record is stale after partition-split,
// which splits the partition into two halves.
// The stale record belongs to the other half.
bool check_if_stale_split_data(const rocksdb::Slice &key) const
```
##########
File path: src/base/pegasus_const.h
##########
@@ -62,4 +62,6 @@ extern const std::string PEGASUS_CLUSTER_SECTION_NAME;
extern const std::string ROCKSDB_ENV_SLOW_QUERY_THRESHOLD;
extern const std::string ROCKSDB_ITERATION_THRESHOLD_TIME_MS;
+
+extern const std::string SPLIT_VALIDATE_PARTITION_HASH;
Review comment:
```suggestion
extern const std::string SPLIT_CHECK_PARTITION_HASH;
```
##########
File path: src/server/pegasus_server_impl.cpp
##########
@@ -2438,6 +2442,25 @@ void pegasus_server_impl::update_rocksdb_iteration_threshold(
}
}
+void pegasus_server_impl::update_validate_partition_hash(
Review comment:
```suggestion
void pegasus_server_impl::update_check_partition_hash(
```
Be consistent with the concept "check_partition_hash", it would be ambiguous.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] hycdong commented on a change in pull request #698: feat(split): gc useless data after partition split
Posted by GitBox <gi...@apache.org>.
hycdong commented on a change in pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698#discussion_r598578204
##########
File path: src/base/pegasus_const.h
##########
@@ -62,4 +62,6 @@ extern const std::string PEGASUS_CLUSTER_SECTION_NAME;
extern const std::string ROCKSDB_ENV_SLOW_QUERY_THRESHOLD;
extern const std::string ROCKSDB_ITERATION_THRESHOLD_TIME_MS;
+
+extern const std::string SPLIT_VALIDATE_PARTITION_HASH;
Review comment:
Actually, the app envs is `replica.split.validate_partition_hash`, so I rename all `check_partition_hash` into `validate_partition_hash`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] hycdong closed pull request #698: feat(split): gc useless data after partition split
Posted by GitBox <gi...@apache.org>.
hycdong closed pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org
[GitHub] [incubator-pegasus] hycdong commented on a change in pull request #698: feat(split): gc useless data after partition split
Posted by GitBox <gi...@apache.org>.
hycdong commented on a change in pull request #698:
URL: https://github.com/apache/incubator-pegasus/pull/698#discussion_r598578204
##########
File path: src/base/pegasus_const.h
##########
@@ -62,4 +62,6 @@ extern const std::string PEGASUS_CLUSTER_SECTION_NAME;
extern const std::string ROCKSDB_ENV_SLOW_QUERY_THRESHOLD;
extern const std::string ROCKSDB_ITERATION_THRESHOLD_TIME_MS;
+
+extern const std::string SPLIT_VALIDATE_PARTITION_HASH;
Review comment:
Actually, the app envs is called `replica.split.validate_partition_hash`, so I rename all `check_partition_hash` into `validate_partition_hash`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@pegasus.apache.org
For additional commands, e-mail: dev-help@pegasus.apache.org