[jira] [Updated] (CAMEL-25770) camel-azure-storage-blob, camel-azure-storage-datalake, camel-google-storage, camel-minio - one listed object that cannot be got (access denied, archived) fails every poll and the consumer never consumes the bucket again

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-25770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen updated CAMEL-25770:
--------------------------------
    Fix Version/s: 4.24.0

> camel-azure-storage-blob, camel-azure-storage-datalake, camel-google-storage, camel-minio - one listed object that cannot be got (access denied, archived) fails every poll and the consumer never consumes the bucket again
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25770
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25770
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-azure, camel-google-storage, camel-minio
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> The same defect as CAMEL-25558 (camel-aws2-s3) and CAMEL-25760 (camel-alibaba-oss, camel-huaweicloud-obs) in four more object-store consumers. Each lists the whole bucket / container / file system at every poll, so one listed item whose get fails every time fails every poll and the consumer is stuck for good. The fix in each module mirrors CAMEL-25558: skip and report the item, fail the poll only when none of the listed items could be got. One pull request per module.
> h3. camel-azure-storage-blob
> {{BlobConsumer.poll}} (main e13303ef1ef7) lists the whole container at every poll ({{createBatchExchangesFromContainer}}, lines 124-145; {{listBlobs}} streams every page) and gets every listed blob in {{addExchange}} (lines 148-168; the get is {{BlobOperations.getBlob}}, or the download to {{fileDir}}). A blob that no longer exists (404) is skipped since CAMEL-25551; any other failure of the get is rethrown, and only a 404 of the container is turned into an empty poll (line 93):
> {code:java}
> try {
>     for (BlobItem blobItem : blobs) {
>         addExchange(blobItem.getName(), blobContainerClient, exchanges, true);
>     }
> } catch (Exception e) {
>     // none of the blobs of this poll is processed
>     releaseNotProcessed(exchanges);
>     throw e;
> }
> {code}
> As the next poll lists the whole container again, one blob whose get fails every time stops the consumer from consuming any blob, for good. The most common case is a blob in the archive tier: a lifecycle management rule moves old blobs to the archive tier, and reading an archived blob fails with 409 BlobArchived until it is rehydrated. Others: access denied for that blob (an ABAC condition on blob index tags), a blob encrypted with a customer-provided key. The consumer keeps running and logs one WARN per poll.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed object whose get fails is skipped for that poll and reported to the exception handler with its key, the other listed objects are consumed, and the poll fails only when none of the listed objects could be got. CAMEL-25760 does the same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{BlobConsumer.createBatchExchangesFromContainer}}: a failed get of one listed blob is collected, the blob is skipped and reported to the consumer's exception handler ("Error getting blob <name> from container <container>. The blob is skipped for this poll."), and the other blobs are consumed; when none of the listed blobs could be got, the poll fails with the first failure as before. A failure with {{blobName}} set, and a failure of the listing, are not changed. Upgrade note in the 4.24 guide.
> *Tests*: new {{BlobConsumerBlobThatCannotBeGotTest}} (2 tests, the mocked {{BlobServiceClient}} of {{BlobConsumerInProgressTest}}; the get of an archived blob throws a {{BlobStorageException}} with status 409). {{blobThatCannotBeGotDoesNotBlockTheOtherBlobs}} (a.txt, b.txt archived, c.txt) fails on main: {{mock://result Received message count. Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedBlobCanBeGot}} passes on main and with the fix; it guards the case the fix must not swallow. camel-azure-storage-blob unit tests: 66 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3) across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try block and rethrows the first failure). The JUnit tests above are the evidence; the 409 BlobArchived answer is from the Azure Storage documentation and was not run against Azure.
> *Affected*: main and camel-4.23.x (the same rethrow); camel-4.22.x and camel-4.18.x have no per-blob catch at all, so there a blob deleted after the listing fails the poll too (read, not run).
> h3. camel-azure-storage-datalake
> {{DataLakeConsumer.poll}} (main e13303ef1ef7, lines 54-79) lists the whole file system at every poll ({{createBatchExchangesFromPath}}, lines 82-101) and gets every listed file with {{createExchangeFromFile}} (lines 103-122; the get is {{DataLakeFileOperations.getFile}}, or the download to {{fileDir}}). Nothing catches the failure of one file:
> {code:java}
> for (PathItem pathItem : items) {
>     if (!pathItem.isDirectory()) {
>         exchanges.add(createExchangeFromFile(pathItem.getName(), dataLakeFileSystemClient));
>     }
> }
> {code}
> A {{DataLakeStorageException}} with status 404 (a file deleted after the listing) ends the whole poll with no exchange (lines 71-77; the exchanges already created for the files listed before it are dropped, not routed), and any other failure fails the poll. As the next poll lists the file system again, one file whose get fails every time stops the consumer from consuming any file, for good. In Data Lake Storage Gen2 every file has its own POSIX ACL, so a file the consumer's identity may list but not read (403 AuthorizationPermissionMismatch) is a normal state, as is a file encrypted with a customer-provided key. The consumer keeps running and logs one WARN per poll.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed object whose get fails is skipped for that poll and reported to the exception handler with its key, the other listed objects are consumed, and the poll fails only when none of the listed objects could be got. CAMEL-25760 does the same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{DataLakeConsumer.createBatchExchangesFromPath}}: a failed get of one listed file is collected, the file is skipped and reported to the consumer's exception handler ("Error getting file <name> from file system <fs>. The file is skipped for this poll."), and the other files are consumed; a file deleted after the listing (404) is skipped without failing the poll; when none of the listed files could be got, the poll fails with the first failure as before. A failure with {{fileName}} set, and a failure of the listing, are not changed. Upgrade note in the 4.24 guide.
> *Tests*: new {{DataLakeConsumerFileThatCannotBeGotTest}} (3 tests, a mocked {{DataLakeServiceClient}}; the read of a forbidden file throws a {{DataLakeStorageException}} with status 403). {{fileThatCannotBeGotDoesNotBlockTheOtherFiles}} (a.txt, b.txt forbidden, c.txt) fails on main: {{mock://result Received message count 0, expected at least 2}}. {{fileDeletedAfterTheListingIsSkipped}} (a.txt, b.txt answering 404, c.txt; nothing is reported) fails on main the same way. {{pollFailsWhenNoListedFileCanBeGot}} passes on main and with the fix; it guards the case the fix must not swallow. camel-azure-storage-datalake unit tests: 19 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3) across the object-store consumers, rule {{listing-poll-uncaught-item-get}} (a Semgrep rule: a {{ScheduledBatchPollingConsumer}} that creates the exchange of every listed item, which downloads it, with no per-item catch). The JUnit tests above are the evidence; the 403 answer is from the Azure Storage documentation and was not run against Azure.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same loop; read, not run).
> h3. camel-google-storage
> {{GoogleCloudStorageConsumer.poll}} (main e13303ef1ef7, lines 122-151) lists the whole bucket at every poll ({{page.iterateAll()}}) and downloads every listed object in {{createExchanges}} / {{addExchange}} (lines 160-209; the download is {{blob.downloadTo}} in {{populateExchange}}, lines 392 and 401). An object that no longer exists (404) is skipped since CAMEL-25519, and an object whose name is rejected as a local path is skipped; any other failure of the download is rethrown and fails the poll:
> {code:java}
> } catch (Exception e) {
>     LOG.warn("Error getting object due: {}", e.getMessage(), e);
>     // none of the objects of this poll is processed
>     releaseInProgress(answer);
>     throw e;
> }
> {code}
> As the next poll lists the whole bucket again, one object whose download fails every time (access denied for that object, an object encrypted with a customer-supplied key (CSEK), an object whose Cloud KMS key version was disabled) stops the consumer from consuming any object of the bucket, for good. The consumer keeps running and logs a WARN with a stack trace per poll, until someone removes the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed object whose get fails is skipped for that poll and reported to the exception handler with its key, the other listed objects are consumed, and the poll fails only when none of the listed objects could be got. CAMEL-25760 does the same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{GoogleCloudStorageConsumer.createExchanges}}: a failed download of one listed object is collected, the object is skipped and reported to the consumer's exception handler ("Error getting object <name> from bucket <bucket>. The object is skipped for this poll."), and the other objects are consumed; when none of the listed objects could be got, the poll fails with the first failure as before. A failure of the listing, and a failure with {{objectName}} set, still fail the poll. Upgrade note in the 4.24 guide.
> *Tests*: new {{ConsumerObjectThatCannotBeGotTest}} (2 tests, with the module's in-process {{LocalStorageHelper}} storage, which can now refuse the reads of an object with 403). {{objectThatCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt refused, c.txt) fails on main: {{mock://result Received message count. Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the fix; it guards the case the fix must not swallow. camel-google-storage unit tests: 63 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3) across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try block and rethrows the first failure). The JUnit tests above are the evidence; the status codes of Cloud Storage were not run against the service.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same rethrow in {{createExchanges}}; read, not run).
> h3. camel-minio
> {{MinioConsumer}} (main e13303ef1ef7) stats every listed object when it creates the exchanges ({{createExchanges}} / {{addExchange}}, lines 201-254, the stat is {{MinioEndpoint.getObjectStat}}), and gets the body of each object when it routes the batch ({{processBatch}}, lines 289-331). An object that no longer exists is skipped since CAMEL-25512. Any other failure is rethrown:
> {code:java}
> } catch (Exception e) {
>     ...
>     LOG.warn("Error getting MinioObject due: {}", e.getMessage());
>     // this and the remaining exchanges of the batch are not processed
>     releaseInProgress(srcObjectName, exchanges);
>     throw e;
> }
> {code}
> So one object whose stat or get fails every time (access denied for that object by a bucket or IAM policy, an object encrypted with a customer key (SSE-C) other than the one of the endpoint) fails every poll: a failed stat drops every exchange of the poll, and a failed get stops the batch, so the objects listed after it are not consumed. The next poll lists the same objects again (with the default {{deleteAfterRead=true}} the objects before it are gone, so the bad object comes first and nothing after it is ever consumed). The consumer keeps running and logs a WARN at every poll; it lasts until someone removes the object.
> camel-aws2-s3 had the same behaviour and CAMEL-25558 changed it: a listed object whose get fails is skipped for that poll and reported to the exception handler with its key, the other listed objects are consumed, and the poll fails only when none of the listed objects could be got. CAMEL-25760 does the same for camel-alibaba-oss and camel-huaweicloud-obs.
> *Fix*: the same in {{MinioConsumer}}, for both places: a failed stat (in {{createExchanges}}) or get (in {{processBatch}}) of one listed object is collected, the object is skipped (released from the in-progress set) and reported to the consumer's exception handler ("Error getting object <name> from bucket <bucket>. The object is skipped for this poll."), and the other objects are consumed; when none of the objects could be got, the poll fails with the first failure as before. A failure of the listing itself, and a failure with {{objectName}} set, still fail the poll. Upgrade note in the 4.24 guide.
> *Tests*: new {{MinioConsumerObjectThatCannotBeGotTest}} (3 tests, the real Minio client against the in-process fake S3 server of the module's tests, which can now refuse the stats and gets of an object with 403 AccessDenied). {{objectWhoseBodyCannotBeGotDoesNotBlockTheOtherObjects}} (a.txt, b.txt whose get is refused, c.txt) fails on main: {{mock://result Received message count. Expected: <2> but was: <1>}} (c.txt is never consumed). {{objectThatCannotBeStatedDoesNotBlockTheOtherObjects}} fails on main: {{Expected: <2> but was: <0>}}. {{pollFailsWhenNoListedObjectCanBeGot}} passes on main and with the fix; it guards the case the fix must not swallow. camel-minio unit tests: 10 run, 0 failures.
> *How it was found*: variant analysis of CAMEL-25558 (the fix of camel-aws2-s3) across the object-store consumers, rule {{listing-poll-rethrows-one-item}} (a Semgrep rule: a {{ScheduledBatchPollingConsumer}} that gets every listed item inside one try block and rethrows the first failure). The JUnit tests above are the evidence; the S3 status codes are those of the S3 API and were not run against a MinIO server.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same rethrow in {{createExchanges}} and {{processBatch}}; on 4.22.x and 4.18.x, which do not have CAMEL-25512, an object deleted after the listing fails the poll too; read, not run). The component is deprecated since 4.23 (CAMEL-24750) but still shipped; CAMEL-25512 was fixed in it after the deprecation.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)