You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pegasus.apache.org by yu...@apache.org on 2021/05/27 10:48:48 UTC
[incubator-pegasus] branch master updated: doc: add rfc about user
specified compaction (#738)
This is an automated email from the ASF dual-hosted git repository.
yuchenhe pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pegasus.git
The following commit(s) were added to refs/heads/master by this push:
new c1848c8 doc: add rfc about user specified compaction (#738)
c1848c8 is described below
commit c1848c87eb875ccccf5904d171adc7c727ca4371
Author: zhao liwei <zl...@163.com>
AuthorDate: Thu May 27 18:48:41 2021 +0800
doc: add rfc about user specified compaction (#738)
---
rfcs/2021-05-27-user-specified-compaction.md | 40 ++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/rfcs/2021-05-27-user-specified-compaction.md b/rfcs/2021-05-27-user-specified-compaction.md
new file mode 100644
index 0000000..62caa21
--- /dev/null
+++ b/rfcs/2021-05-27-user-specified-compaction.md
@@ -0,0 +1,40 @@
+# User Specified Compaction
+
+## Summary
+
+In Pegasus, sometimes we should add user specified compaction policy to reduce disk usage. This RFC proposes a user specified compaction design.
+
+## Design
+
+Add two classes named with `compaction_operation` and `compaction_rule`.
+
+`compaction_filter_rule` represents the compaction rule to filter the keys which are stored in rocksdb.
+There are three types of compaction operation:
+- Hashkey rule, which supports prefix match, postfix match and anywhere match.
+- Sortkey rule. Just like hashkey rule, it also supports prefix match, postfix match and anywhere match.
+- TTL rule. It supports time range match with format [begin_ttl, end_ttl]
+
+`compaction_operation` represents the compaction operation. A compaction operation will be executed when all the corresponding compaction rules are matched.
+There are two types of compaction filter rule:
+- Delete. It represents that we should delete this key when all the rules are matched.
+- Update TTL. It represents that we should update TTL when all the rules are matched.
+
+Finally, we should save the information about user specified compaction in app env. In order to make these information can still be retrieved after the machine is restarted.
+
+## Class Diagram
+
+Here is the class diagram for user specified compaction.
+
+```
+ +---------------+ +-----------------+
+ | compaction op +------------------------------+ compaction rule |
+ +------^--------+ +--------^--------+
+ | |
+ _________|_________ ______________________|______________________
+ | | | | |
+ | | | | |
+ | | | | |
++------------+ +------------+ +--------------+ +--------------+ +--------------+
+| update ttl | | delete | | hashkey rule | | sortkey rule | | ttl rule |
++------------+ +------------+ +--------------+ +--------------+ +--------------+
+```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pegasus.apache.org
For additional commands, e-mail: commits-help@pegasus.apache.org