[jira] [Commented] (JAMES-3570) Errors running deduplicated BlobStore on top of OVH (Swift) S3 APIs

["Benoit Tellier (Jira)" <se...@james.apache.org>]

    [ https://issues.apache.org/jira/browse/JAMES-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350840#comment-17350840 ] 

Benoit Tellier commented on JAMES-3570:
---------------------------------------

We successfully setted up this approach within our TMail product.

We noticed a nice p99 / mean time reduction upon email creation.

See https://github.com/linagora/tmail-backend/pull/152#issue-650601889 for a bench result.

> Errors running deduplicated BlobStore on top of OVH (Swift) S3 APIs
> -------------------------------------------------------------------
>
>                 Key: JAMES-3570
>                 URL: https://issues.apache.org/jira/browse/JAMES-3570
>             Project: James Server
>          Issue Type: Bug
>          Components: Blob
>    Affects Versions: 3.6.0
>            Reporter: Benoit Tellier
>            Priority: Major
>              Labels: bug
>
> h3. What 
> At Linagora, we deploy our James offer at OVH, which provides a Swift backed S3 API.
> Running the deduplicating blob store on such a setup, we encountered the following errors:
>  - 400 errors with no messages upon PUT
>  - NoSuckKeyExceptions being thrown by the S3 driver
> We noticed these exceptions to be occasionally thrown by the post processing (event bus), for emails with several recipients. Post processing eventually succeeds.
> In addition to unpleasant errors in the log, we wonder if two concurrent APPENDs could not lead to possible data corruption...
> h3. Why
> Swift do not implement strong read/write isolation. We did run the James BlobStore test suite against OVH S3 APIs, and encountered the same errors.
> We thus link these errors to a default of isolation.
> h3. How to fix
> Our strategy so far is to implement https://github.com/apache/james-project/blob/master/src/adr/0015-objectstorage-blobid-list.md, but as part of a Linagora product built on top of James (because the following ADR had been controversial and rejected by a part of the community).
> The idea behind this fix is to leverage Cassandra to avoid, most of the time, to store a blob twice.
> We are not against contributing this mitigation under Apache, but we wish to fix it timely on our side first. Please show interest if we want this to be shared.
> Amongst the alternatives we can consider:
>  - To run on an alternative object storage (Zenko cloud server, used by our test suite for instance, AWS s3). 
>  - To check the object store itself before APPENDs. It does not prevent data races (as the blob might be inserted by another thread between
> the check and the append). Also previous experiments have shown it has a great cost on performance. CF https://github.com/linagora/james-project/pull/2011
>  - To disable deduplication. This allows to effectively delete data (not yet the case for deduplication) but will result in data duplication and 
> thus a storage cost increase. Can be done today via a configuration option...
>  h3. Interaction with other features
> An effort regarding implementing deduplication is currently on pause. BlobId will ship a notion of generation, we never delete blobs of the current generation
> (and of a buffer generation to prevent clock skew), so we never delete blobs that we are appending, so by design the de-duplicated content removal will not 
> interfere with this blobIdList mechanism.
> This blobId list mechanism is scoped to the default bucket not to interfere with the DeletedMessageVault.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org