[jira] [Created] (JAMES-4234) BlobMailRepository: Isolate removeAll across mail repositories to prevent catastrophic data loss and cross-repository deletion

["ilya terskov (Jira)" <se...@james.apache.org>]

ilya terskov created JAMES-4234:
-----------------------------------

             Summary: BlobMailRepository: Isolate removeAll across mail repositories to prevent catastrophic data loss and cross-repository deletion
                 Key: JAMES-4234
                 URL: https://issues.apache.org/jira/browse/JAMES-4234
             Project: James Server
          Issue Type: Bug
          Components: Blob
    Affects Versions: master
            Reporter: ilya terskov


https://github.com/apache/james-project/pull/3213

Description
Context & Problem Statement
In BlobMailRepository, multiple mail repositories may share the same underlying BlobStore and bucket. Previously, several critical issues existed in repository isolation, cleanup, and life-cycle management:

Unbounded removeAll() Deletions (Cross-Repository Data Loss):
removeAll() called listBlobs() across the entire bucket without scoping to the repository's path prefix. As a result, invoking removeAll() on a single repository wiped out metadata and MIME blobs belonging to all other mail repositories sharing the same bucket.
Missing Ownership & Cross-Repository MIME Reference Protection:
Operations blindly trusted metadata and MIME blob references. If a corrupted, forged, or cross-referenced metadata entry pointed to MIME blobs belonging to a different repository, operations like remove() or removeAll() could delete foreign MIME data, and retrieve() could leak foreign emails.
MIME Blob Leaks on Mail Overwrite:
Calling store(mail) with an existing MailKey overwrote the metadata entry with new MIME blob IDs but left the superseded header and body MIME blobs orphaned in the blobstore.
Stream Leaks & Corrupted Metadata Resiliency:
Metadata decoding did not reliably close underlying streams via Using.resource. Furthermore, non-JSON or schema-mismatched blobs encountered during enumeration or retrieval could crash operations rather than being gracefully skipped.
Proposed Changes
1. Path-Scoped Listing & Verification (ownedMails and readOwned)
Scoped metadata blob listing to the repository's prefix using listBlobs(bucket, url.getPath.asString() + "/").
Implemented readOwned(id: MailPartsId) as a centralized gatekeeper for all operations (retrieve, remove, removeAll, store, list, size):
Validates that the metadata payload's mail name matches the expected blob ID (mailMetadataBlobIdFactory.of(mail.getName) == id.metadataBlobId).
Enforces isOwnedMimePart: verifies that both headerBlobId and bodyBlobId reside strictly under the repository's expected MIME path prefix (<repoPath>/mimeMessagedata/). Any foreign MIME references are rejected.
Gracefully ignores missing (ObjectNotFoundException), malformed (JsonParseException), or invalid schema (JsResultException) metadata entries.
2. Cross-Repository Isolation & Progress Tracking
Implemented removeAll(progressCallback: Consumer[MailKey]) and updated removeAll() to iterate exclusively over ownedMails(), deleting MIME parts followed by metadata blobs, and reporting deleted keys via callback.
Fixed size and list to count/iterate only valid, owned mails.
Updated remove(key) to delete blobs only after ownership and MIME prefix validation pass.
3. Cleanup of Superseded MIME Blobs on Overwrite
In store(mc: Mail), checked for existing owned metadata before persisting the new mail.
If overwriting an existing key, the superseded MIME blobs are asynchronously deleted with a backoff retry (Retry.backoff(2, Duration.ofMillis(50))). If deletion exhausts retries, a warning is logged with the specific blob IDs, preventing silent data leakage while keeping the primary store operation reliable.
4. Safe Resource Handling
Wrapped Store.CloseableByteSource stream decoding in Using.resource within MailMetadataDecoder to guarantee stream closure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: server-dev-unsubscribe@james.apache.org
For additional commands, e-mail: server-dev-help@james.apache.org