[jira] [Created] (CAMEL-25558) camel-aws2-s3 - a listed object that cannot be got (deleted meanwhile, or a failed get) fails the poll and stays in the in-progress repository, so it is never consumed until a restart

["shashank (Jira)" <ji...@apache.org>]

shashank created CAMEL-25558:
--------------------------------

             Summary: camel-aws2-s3 - a listed object that cannot be got (deleted meanwhile, or a failed get) fails the poll and stays in the in-progress repository, so it is never consumed until a restart
                 Key: CAMEL-25558
                 URL: https://issues.apache.org/jira/browse/CAMEL-25558
             Project: Camel
          Issue Type: Bug
          Components: camel-aws2-s3
            Reporter: shashank
            Assignee: shashank


When the consumer lists the bucket, {{AWS2S3Consumer.createExchanges(List)}} adds each listed key to the endpoint's {{inProgressRepository}} (line 264 at main 3a6cdba883a5) and then gets the object (line 287). When that get throws, the catch block (lines 301-315) closes the streams and removes the keys of the exchanges that were already created, and rethrows, so the poll fails. The key of the object that failed has no exchange yet, so it is not removed. The default repository is a memory repository that is only cleared when the endpoint stops, so every later poll skips that key as "already in progress" (lines 264-268).

Effects:
* A listed object that is deleted before the get (another consumer on the same bucket, a lifecycle rule, or the exchange of an earlier poll that completes after the listing, with an asynchronous route) makes the get fail with {{NoSuchKeyException}}: that poll fails. The other listed objects are consumed by a following poll (with a truncated listing the marker has already moved on, lines 173-188, so they wait until the listing starts over). A new object stored later under the same key is never consumed.
* An object whose get fails for another reason (403, throttling, a network error), or whose download fails while it is read into the body ({{includeBody=true}}, line 500), fails that poll in the same way, stays in the bucket, and is never consumed again, until the endpoint is restarted. In the second case the exchange that was created for it (line 493) is also never released to the exchange factory.

This is not new and not caused by CAMEL-25521: camel-4.22.x has the same loop and catch block.

For comparison: camel-minio (CAMEL-25512) and camel-google-storage (CAMEL-25519) skip a listed object that no longer exists (DEBUG) and consume the rest; camel-ibm-cos skips a listed object whose get fails for any reason (WARN with the key and the message, no stack trace), removes its key and continues with the rest.

h3. Reproduction

New {{AWS2S3ConsumerDeletedObjectTest}} (6 tests) with the in-process {{FakeS3Server}} of the module (it can now remove an object right after a listing returned it, refuse the gets of an object with 403, and cut the body of a get short), and {{AWS2S3ConsumerDeletedObjectPooledExchangeTest}}, which runs the same 6 with {{PooledExchangeFactory}} and adds one. The scheduler is not started ({{startScheduler=false}}); the tests call {{poll()}}. On main all 13 fail:
* {{listedObjectThatNoLongerExistsIsSkipped}}: {{a.txt}}, {{b.txt}}, {{c.txt}}; {{b.txt}} is removed after the listing. Expected: the poll returns 2 and routes a and c with {{CamelBatchSize}} 2. On main: {{NoSuchKeyException: The specified key does not exist}} from {{poll()}}.
* {{objectCreatedAgainAfterItWasSkippedIsConsumed}}: after that poll a new {{b.txt}} is stored and must be consumed by the next poll. On main: {{NoSuchKeyException}} from the first poll.
* {{objectWhoseGetIsRefusedDoesNotStopTheOtherObjects}}: every get of {{b.txt}} is refused. Expected: the poll returns 2 (a and c), the next poll consumes a new {{d.txt}} and tries {{b.txt}} again, and {{b.txt}} is consumed once its get is allowed. On main: {{S3Exception: Access Denied}} from the first {{poll()}}.
* {{skippedObjectOfATruncatedListingIsTriedAgainWhenTheListingStartsOver}} ({{maxMessagesPerPoll=2}}, four objects, {{b.txt}} refused): the polls return 1, 2 (second page, {{b.txt}} not tried), 0 (listing starts over, {{b.txt}} tried again), then 1 once the get is allowed. On main: {{S3Exception}} from the first {{poll()}}.
* {{objectWhoseDownloadFailsIsSkippedAndNotLeftInProgress}}: the content stream throws an {{IOException}} while it is read into the body. Expected: the poll returns 0, the stream is closed, the key is not in progress. On main: {{RuntimeCamelException}} ({{IOException: connection reset}}) from {{poll()}}.
* {{objectStreamIsClosedWhenIncludeS3ObjectFails}}: an overridden {{includeS3Object}} throws. Expected as above. On main: {{IllegalStateException}} from {{poll()}}.
* {{exchangeOfAnObjectWhoseDownloadFailsIsGivenBackToThePool}} (pooled): the body of {{b.txt}} is cut short. Expected: a, c and d are routed with their own bodies, 4 exchanges taken from the pool and 4 released. On main: {{RuntimeCamelException}} ({{ConnectionClosedException: Premature end of Content-Length delimited message body}}) from {{poll()}}.

h3. Proposed fix

In {{createExchanges(List)}}: a failure to get one listed object, or to create its exchange, no longer fails the poll. The object is skipped (stream closed, exchange given back to the exchange factory, key removed from the in-progress repository), the other listed objects are consumed in the same poll, the skipped object is not counted, and a later poll tries it again. {{NoSuchKeyException}} is logged at DEBUG; any other failure at WARN with the key and the message, without a stack trace, because an object that can never be read (archived, access denied) is logged at every poll that lists it. This is what camel-ibm-cos does; the alternative (keep failing the poll and only remove the key) would make such an object fail every poll and block the objects listed after it.

Consequence: such a failure is no longer a failed poll, so it does not reach the exception handler ({{bridgeErrorHandler}}) and does not count for {{backoffErrorThreshold}}. The listing marker is set from the listing response before the gets, as before: with a truncated listing a skipped object is tried again when the listing starts over, otherwise at the next poll.

Not changed, known limits: with {{fileName}}, a failed get still fails the poll. An {{Error}} (for example {{OutOfMemoryError}} while a large object is read into the body) is not caught, so its key still stays in progress.

With the pooled exchange factory the exchange of a skipped object is given back with {{done()}}: {{releaseExchange(exchange, true)}} on an exchange that is not done adds it to the pool twice, and two later objects of the poll then share it (seen with the pooled test). The same call for never-routed exchanges exists on main in this consumer ({{processBatch}} when stopping, the outer catch) and in other consumers; not changed here.

No new option. 4.23 upgrade-guide note under the existing camel-aws2-s3 in-progress entry. camel-aws2-s3 unit tests with the fix: 98, 0 failures (the 68 integration tests need LocalStack and were not run).

Found by comparing the consumer with the other object storage consumers after CAMEL-25521, then confirmed with the real consumer as above.

Affected: main; camel-4.22.x, camel-4.18.x and camel-4.14.x have the same loop and catch block (read, not run).

Duplicate check (2026-10-11): JIRA text "aws2-s3 NoSuchKey" (1: CAMEL-25521, the related fix, which does not change this path), "AWS2S3Consumer NoSuchKeyException" (1: CAMEL-17110, which added the in-progress repository), component camel-aws2-s3 with text "inProgressRepository" (1: CAMEL-25521), component camel-aws2-s3 created since 2026-01-01 (10 issues, none about a deleted object or a failed get in the consumer); GitHub pull requests "AWS2S3Consumer NoSuchKey" and "aws2-s3 NoSuchKey": none; no open pull request touches {{AWS2S3Consumer}}.

_Filed with Claude Code on behalf of allthingssecurity._




--
This message was sent by Atlassian Jira
(v8.20.10#820010)