You are viewing a plain text version of this content. The canonical link for it is here.
Posted to server-dev@james.apache.org by "Benoit Tellier (Jira)" <se...@james.apache.org> on 2021/08/16 08:08:00 UTC
[jira] [Comment Edited] (JAMES-3150) Implement Garbage Colletion
for blobs
[ https://issues.apache.org/jira/browse/JAMES-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17399583#comment-17399583 ]
Benoit Tellier edited comment on JAMES-3150 at 8/16/21, 8:07 AM:
-----------------------------------------------------------------
As describe in https://github.com/apache/james-project/pull/594 we propose a simpler to implement deduplication method, targeting middle size deployments.
The steps would be:
- 1. Provide a way for BlobStores to list blobIds in a bucket
- 2. Provide an interface to list blob references, and implement it for each entity referencing blobs
- 3. Provide a GenerationAware BlobId factory - generation solves concurrency issues and can be generated without synchronisation
- 4. Write a simple algorithm leveraging bloom filters performing the GC.
A (quick) proof of concept can be found here: https://github.com/chibenwa/james-project/tree/dedup-poc
Limits and tweaks of this approach are defined in the ADR.
Doubts I express regarding the previous (unfinished) development efforts are also listed there. But please note that I believe the two approach can co-exist, and that one could be choosing the algorithm she wishes to apply, might a benevolent person decide to finish the iteration based approach that had been started but not finished.
was (Author: btellier):
As describe in https://github.com/apache/james-project/pull/594 we propose a simpler to implement deduplication method, targeting middle size deployments.
The steps would be:
- 1. Provide a way for BlobStores to list blobIds in a bucket
- 2. Provide an interface to list blob references, and implement it for each entity referencing blobs
- 3. Provide a GenerationAware BlobId factory - generation solves concurrency issues and can be generated without synchronisation
- 4. Write a simple algorithm leveraging bloom filters performing the GC.
Limits and tweaks of this approach are defined in the ADR.
Doubts I express regarding the previous (unfinished) development efforts are also listed there. But please note that I believe the two approach can co-exist, and that one could be choosing the algorithm she wishes to apply, might a benevolent person decide to finish the iteration based approach that had been started but not finished.
> Implement Garbage Colletion for blobs
> -------------------------------------
>
> Key: JAMES-3150
> URL: https://issues.apache.org/jira/browse/JAMES-3150
> Project: James Server
> Issue Type: Improvement
> Components: Blob
> Affects Versions: 3.3.0
> Reporter: Gautier DI FOLCO
> Priority: Major
>
> With the blob store deduplication, dropping a blob in a distributed environment is impossible if we want to keep an acceptable concurrency level.
> A Garbage Collector should be created in order to drop old blobs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org