You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@james.apache.org by bt...@apache.org on 2021/09/06 04:30:54 UTC
[james-project] branch master updated: JAMES-3150 Document blob
garbage collection (#631)
This is an automated email from the ASF dual-hosted git repository.
btellier pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/james-project.git
The following commit(s) were added to refs/heads/master by this push:
new 4130a60 JAMES-3150 Document blob garbage collection (#631)
4130a60 is described below
commit 4130a60e5f429860b8673ebbc351cd6015ebfb5d
Author: Benoit TELLIER <bt...@linagora.com>
AuthorDate: Mon Sep 6 11:30:46 2021 +0700
JAMES-3150 Document blob garbage collection (#631)
Co-authored-by: Rene Cordier <re...@gmail.com>
---
.../pages/distributed/architecture/index.adoc | 6 ++--
.../pages/distributed/configure/blobstore.adoc | 15 ++++++--
.../pages/distributed/operate/webadmin.adoc | 39 ++++++++++++++++++++
src/site/markdown/server/manage-webadmin.md | 41 ++++++++++++++++++++++
src/site/xdoc/server/config-blobstore.xml | 14 ++++++--
5 files changed, 106 insertions(+), 9 deletions(-)
diff --git a/docs/modules/servers/pages/distributed/architecture/index.adoc b/docs/modules/servers/pages/distributed/architecture/index.adoc
index fc0ae77..3eeea53 100644
--- a/docs/modules/servers/pages/distributed/architecture/index.adoc
+++ b/docs/modules/servers/pages/distributed/architecture/index.adoc
@@ -277,11 +277,11 @@ Supported backends include S3 compatible ObjectStorage (link:https://wiki.openst
Encryption can be configured on top of ObjectStorage.
-Blobs are currently deduplicated in order to reduce storage space. This means that two blobs with
+Blobs can currently be deduplicated in order to reduce storage space. This means that two blobs with
the same content will be stored one once.
-The downside is that deletion is more complicated, and a garbage collection needs to be run. This is a work
-in progress. See link:https://issues.apache.org/jira/browse/JAMES-3150[JAMES-3150].
+The downside is that deletion is more complicated, and a garbage collection needs to be run. A first implementation
+based on bloom filters can be used and triggered using the WebAdmin REST API.
=== Task Manager
diff --git a/docs/modules/servers/pages/distributed/configure/blobstore.adoc b/docs/modules/servers/pages/distributed/configure/blobstore.adoc
index 73f3d48..e9cd4a0 100644
--- a/docs/modules/servers/pages/distributed/configure/blobstore.adoc
+++ b/docs/modules/servers/pages/distributed/configure/blobstore.adoc
@@ -35,11 +35,20 @@ If you choose to enable deduplication, the mails with the same content will be s
WARNING: Once this feature is enabled, there is no turning back as turning it off will lead to the deletion of all
the mails sharing the same content once one is deleted.
-This feature also requires a garbage collector mechanism to effectively drop blobs, which is not implemented yet.
+NOTE: If you are upgrading from James 3.5 or older, the deduplication was enabled.
-Consequently, all the requested deletions will not be performed, meaning that blobstore will only grow.
+Deduplication requires a garbage collector mechanism to effectively drop blobs. A first implementation
+based on bloom filters can be used and triggered using the WebAdmin REST API. See
+xref:distributed/operate/webadmin.adoc#_running_blob_garbage_collection[Running blob garbage collection].
-NOTE: If you are upgrading from James 3.5 or older, the deduplication was enabled.
+In order to avoid concurrency issues upon garbage collection, we slice the blobs in generation, the two more recent
+generations are not garbage collected.
+
+*deduplication.gc.generation.duration*: Allow controlling the duration of one generation. Longer implies better deduplication
+but deleted blobs will live longer. Duration, defaults on 30 days, the default unit is in days.
+
+*deduplication.gc.generation.family*: Every time the duration is changed, this integer counter must be incremented to avoid
+conflicts. Defaults to 1.
=== Encryption choice
diff --git a/docs/modules/servers/pages/distributed/operate/webadmin.adoc b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
index 282bf08..ff6b52f 100644
--- a/docs/modules/servers/pages/distributed/operate/webadmin.adoc
+++ b/docs/modules/servers/pages/distributed/operate/webadmin.adoc
@@ -2306,6 +2306,45 @@ the following `additionalInformation`:
}
....
+== Running blob garbage collection
+
+When deduplication is enabled one needs to explicitly run a garbage collection in order to delete no longer referenced
+blobs.
+
+To do so:
+
+....
+curl -XDELETE http:ip:port/blobs?scope=unreferenced
+....
+
+link:#_endpoints_returning_a_task[More details about endpoints returning a task].
+
+Additional parameters include Bloom filter tuning parameters:
+
+ - *associatedProbability*: Allow to define the targeted false positive rate. Note that subsequent runs do not have the
+same false-positives.
+ - *expectedBlobCount*: Expected count of blobs used to size the bloom filters.
+
+The created task has the following additional information:
+
+....
+{
+ "referenceSourceCount": 3456,
+ "blobCount": 5678,
+ "gcedBlobCount": 1234,
+ "bloomFilterExpectedBlobCount": 10000,
+ "bloomFilterAssociatedProbability": 0.01
+}
+....
+
+Where:
+
+ - *bloomFilterExpectedBlobCount* correspond to the supplied *expectedBlobCount* query parameter.
+ - *bloomFilterAssociatedProbability* correspond to the supplied *associatedProbability* query parameter.
+ - *referenceSourceCount* is the count of distinct blob references encountered while populating the bloom filter.
+ - *blobCount* is the count of blobs tried against the bloom filter. This value can be used to better size the bloom
+filter in later runs.
+ - *gcedBlobCount* is the count of blobs that were garbage collected.
== Administrating Recipient rewriting
diff --git a/src/site/markdown/server/manage-webadmin.md b/src/site/markdown/server/manage-webadmin.md
index 571260c..25bd7a6 100644
--- a/src/site/markdown/server/manage-webadmin.md
+++ b/src/site/markdown/server/manage-webadmin.md
@@ -51,6 +51,7 @@ Finally, please note that in case of a malformed URL the 400 bad request respons
- [Sending email over webAdmin](#Sending_email_over_webAdmin)
- [Administrating DLP Configuration](#Administrating_DLP_Configuration)
- [Administrating Sieve quotas](#Administrating_Sieve_quotas)
+ - [Running blob garbage collection](#Running_blob_garbage_collection)
- [Administrating jmap uploads](#Administrating_jmap_uploads)
- [Deleted Messages Vault](#Deleted_Messages_Vault)
- [Task management](#Task_management)
@@ -3517,6 +3518,46 @@ Response codes:
- 204: Operation succeeded
+## Running blob garbage collection
+
+When deduplication is enabled one needs to explicitly run a garbage collection in order to delete no longer referenced
+blobs.
+
+To do so:
+
+```
+curl -XDELETE http:ip:port/blobs?scope=unreferenced
+```
+
+[More details about endpoints returning a task](#Endpoints_returning_a_task).
+
+Additional parameters include Bloom filter tuning parameters:
+
+ - **associatedProbability**: Allow to define the targeted false positive rate. Note that subsequent runs do not have the
+same false-positives.
+ - **expectedBlobCount**: Expected count of blobs used to size the bloom filters.
+
+The created task has the following additional information:
+
+```json
+{
+ "referenceSourceCount": 3456,
+ "blobCount": 5678,
+ "gcedBlobCount": 1234,
+ "bloomFilterExpectedBlobCount": 10000,
+ "bloomFilterAssociatedProbability": 0.01
+}
+```
+
+Where:
+
+ - **bloomFilterExpectedBlobCount** correspond to the supplied **expectedBlobCount** query parameter.
+ - **bloomFilterAssociatedProbability** correspond to the supplied **associatedProbability** query parameter.
+ - **referenceSourceCount** is the count of distinct blob references encountered while populating the bloom filter.
+ - **blobCount** is the count of blobs tried against the bloom filter. This value can be used to better size the bloom
+filter in later runs.
+ - **gcedBlobCount** is the count of blobs that were garbage collected.
+
## Administrating Jmap Uploads
- [Cleaning upload repository](#Cleaning_upload_repository)
diff --git a/src/site/xdoc/server/config-blobstore.xml b/src/site/xdoc/server/config-blobstore.xml
index 519701d..97e830e 100644
--- a/src/site/xdoc/server/config-blobstore.xml
+++ b/src/site/xdoc/server/config-blobstore.xml
@@ -53,14 +53,22 @@
The generated startup warning log can be deactivated via the <code>cassandra.blob.store.disable.startup.warning</code> environment
variable being positioned to <code>false</code>.
</dd>
-
<dt><strong>deduplication/enable</strong></dt>
<dd>Mandatory. Supported value: true and false.</dd>
<dd>If you choose to enable deduplication, the mails with the same content will be stored only once.</dd>
<dd>Warning: Once this feature is enabled, there is no turning back as turning it off will lead to the deletion of all</dd>
<dd>the mails sharing the same content once one is deleted.</dd>
- <dd>This feature also requires a garbage collector mechanism to effectively drop blobs, which is not implemented yet.</dd>
- <dd>Consequently, all the requested deletions will not be performed, meaning that blobstore will only grow.</dd>
+ <dd>This feature also requires a garbage collector mechanism to effectively drop blobs. A first implementation
+ based on bloom filters can be used and triggered using the WebAdmin REST API. See
+ <a href="manage-webadmin.html#Running_blob_garbage_collection">Running blob garbage collection</a>.
+ In order to avoid concurrency issues upon garbage collection, we slice the blobs in generation, the two more recent
+ generations are not garbage collected.</dd>
+ <dd><strong>deduplication.gc.generation.duration</strong></dd>
+ <dd>Allow controlling the duration of one generation. Longer implies better deduplication
+ but deleted blobs will live longer. Duration, defaults on 30 days, the default unit is in days.</dd>
+ <dd><strong>deduplication.gc.generation.family</strong></dd>
+ <dd>Every time the duration is changed, this integer counter must be incremented to avoid
+ conflicts. Defaults to 1.</dd>
<dd>Upgrade note: If you are upgrading from James 3.5 or older, the deduplication was enabled.</dd>
</dl>
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@james.apache.org
For additional commands, e-mail: notifications-help@james.apache.org