[jira] [Updated] (CAMEL-25758) camel-azure-storage-blob - consumer with moveAfterRead deletes the source blob although the copy to the destination container failed, was aborted or its status could not be read: the blob is lost

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25758:
--------------------------------
    Fix Version/s: 4.24.0

> camel-azure-storage-blob - consumer with moveAfterRead deletes the source blob although the copy to the destination container failed, was aborted or its status could not be read: the blob is lost
> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25758
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25758
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-azure
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> With {{moveAfterRead=true}} the blob consumer moves a consumed blob in {{BlobConsumer.processCommit}} (main e13303ef1ef7, lines 283-286) as copy + delete:
> {code:java}
> copyBlobToDestination(containerName, blobName, destinationContainer, destinationBlobName);
> deleteBlob(containerName, blobName);
> {code}
> and {{copyBlobToDestination}} (lines 362-363) runs:
> {code:java}
> destBlobClient.beginCopy(sourceBlobClient.getBlobUrl(), Duration.ofSeconds(2))
>         .waitForCompletion(timeout);
> {code}
> The result of {{waitForCompletion}} is not looked at. A Copy Blob is an asynchronous operation of the service: its status is {{pending}} until it ends as {{success}}, {{aborted}} or {{failed}} (Get Blob Properties, {{x-ms-copy-status}}). The {{SyncPoller}} of {{BlobClientBase.beginCopy}} (azure-storage-blob 12.32, read from the jar) does not throw when the copy does not succeed: it returns the last poll response once its status is complete, that is {{FAILED}} for a failed copy, {{USER_CANCELLED}} for an aborted copy, and the custom complete status {{POLLING_FAILED}} when reading the status of a pending copy failed (an exception of {{getProperties()}} is caught in {{onPoll}}). Apart from a copy request that the service refuses ({{BlobStorageException}} when the copy starts), only a timeout throws ({{RuntimeException}} with a {{TimeoutException}}). In all three cases the consumer deletes the source blob next. The destination of a failed or aborted copy has no content (Copy Blob remarks: an aborted copy leaves a blob of 0 bytes, a copy that times out on the service leaves an empty blob), and with {{POLLING_FAILED}} the copy can still be pending when the source is deleted ("any changes to the source while the copy operation is in progress will cause that operation to fail", Copy Blob remarks). The blob is gone from both containers and Camel logs nothing.
> How likely: the move is always within one storage account (one {{BlobServiceClient}}), and Microsoft Learn says such a copy "can complete synchronously". A copy that completes synchronously is read back as {{success}} and is not affected. The loss needs a copy that the service runs asynchronously and that then fails or is aborted, or a failed status read while it is pending. How often same-account copies are asynchronous is not documented.
> The copy was a synchronous {{copyFromUrl}} up to camel-4.18.x (it throws on failure); CAMEL-24158 switched to {{beginCopy}} for the authorization of the copy source.
> *Fix* (one class): {{processCommit}} deletes the source blob only when the copy ended with {{SUCCESSFULLY_COMPLETED}}. Otherwise it reports the copy status to the consumer's exception handler and keeps the blob, which is consumed again on the next poll, the same as after a copy request that fails with a {{BlobStorageException}} today. A copy that keeps failing is therefore retried at every poll with one exception handler call each time, as a copy that is refused already is. A copy reported as {{POLLING_FAILED}} that had in fact succeeded is copied and routed again on the next poll (at least once instead of a possible loss); if the status can never be read (credentials that may write but not read the destination container), every move is reported and retried this way, where main deleted the source without knowing the outcome. A copy that times out is unchanged: the exception leaves the on completion, Camel logs it and the source blob is kept.
> *Tests*: new {{BlobConsumerMoveAfterReadCopyStatusTest}} (4 tests, Mockito clients as in {{BlobConsumerInProgressTest}}; the poller returns the given status): the source blob is kept and the exception handler is called when the copy ends {{FAILED}}, {{USER_CANCELLED}} or {{POLLING_FAILED}}; the source blob is deleted after a successful copy. The three "kept" tests fail on main ({{NeverWantedButInvoked: blobClient.delete()}}), the fourth passes on main and guards the success path of the fix. The camel-azure-storage-blob unit tests pass with the fix (68 run, 0 failures).
> *How it was found*: deterministic simulation testing with a fake Azure storage world (the consumer, its scheduled polls, its on completion and Camel's routing real; the sync {{BlobServiceClient}} / {{BlobContainerClient}} / {{BlobClient}} faked from the Blob service REST reference, with asynchronous copies whose final status the seed picks). The seed picked {{moveAfterRead}} or {{deleteAfterRead}}, {{maxResultsPerPage}}, the poll delay, the {{timeout}}, an asynchronous route step, fault rates, and how often a copy stays pending and ends as failed, aborted or with a failed status read. The invariant "nothing lost: every blob is still in the source container or was moved with its content" failed in 99 of 1000 seeds on main, all with {{moveAfterRead=true}} and a copy that did not succeed (first failing seed 11: a pending copy ended as failed after about 4 s and the source blob was deleted); with the fix, 0 of the same 1000 seeds fail. The world's self-check: with CAMEL-25551 reverted, 733 of the 1000 seeds fail its invariant "a blob is not routed again while its exchange is in flight".
> *Affected*: main, camel-4.23.x and camel-4.22.x (the same {{beginCopy(...).waitForCompletion(timeout)}} followed by the delete; read, not run). camel-4.18.x uses {{copyFromUrl}} and is not affected.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)