[jira] [Created] (CAMEL-25700) camel-aws2-sqs - with extendMessageVisibility=true and maxMessagesPerPoll > 1, the messages that wait in the batch are not extended and are delivered a second time

["shashank (Jira)" <ji...@apache.org>]

shashank created CAMEL-25700:
--------------------------------

             Summary: camel-aws2-sqs - with extendMessageVisibility=true and maxMessagesPerPoll > 1, the messages that wait in the batch are not extended and are delivered a second time
                 Key: CAMEL-25700
                 URL: https://issues.apache.org/jira/browse/CAMEL-25700
             Project: Camel
          Issue Type: Bug
          Components: camel-aws2-sqs
            Reporter: shashank
            Assignee: shashank


With {{extendMessageVisibility=true}} the SQS consumer keeps the messages it is processing invisible: a background task ({{TimeoutExtender}}) calls {{ChangeMessageVisibilityBatch}} for every message it knows about before its visibility timeout runs out.

A poll receives up to {{maxMessagesPerPoll}} messages at once, and their visibility timeout starts at that receive. {{Sqs2Consumer.processBatch}} then routes them one after the other, and registers each message with the extender only when its own turn comes ({{timeoutExtender.add(exchange)}} inside the loop, line 151 on main d46d51c24550). While the first messages are processed (synchronous route), the ones that wait behind them are not extended at all. When the processing of the earlier messages takes longer than {{visibilityTimeout}}, SQS makes the waiting messages visible again and delivers them to another consumer (or to the next poll of this one): they are processed twice, and the delete of the first copy uses a receipt handle that is no longer current.

This is the scenario of CAMEL-10587 (2016, camel-aws): {{maxMessagesPerPoll=10}}, {{extendMessageVisibility=true}}, {{visibilityTimeout=30}}, a slow route, "after 30 seconds only 1 message is in flight". That ticket was closed as Abandoned; its only commit (c6a15565acdf, 2017) started the extender's executor before the first poll; the per-message registration stayed in the loop, and camel-aws2-sqs (and the batched extender of CAMEL-19656) kept it.

Trigger: {{extendMessageVisibility=true}} (the option users set because their processing is slow) with {{maxMessagesPerPoll}} greater than 1 and a synchronous route where the messages before the last one together take longer than {{visibilityTimeout}}. Example: 10 messages per poll, 10 seconds each, {{visibilityTimeout=30}}: at least messages 4 to 10 are delivered again (2 and 3 are extended only at about the moment they expire). With the default {{maxMessagesPerPoll}} (one message per poll) nothing changes.

h3. Reproduction

New {{SqsConsumerExtendMessageVisibilityBatchTest}} with the module's {{AmazonSQSClientMock}} (no AWS): two messages on the queue, {{maxMessagesPerPoll=2&visibilityTimeout=2&extendMessageVisibility=true}}, the route holds the first message until the extender has sent a {{ChangeMessageVisibilityBatch}} entry for the receipt handle of the second (at most 10 seconds). On main it never does:
{noformat}
[the visibility of the second message of the batch is extended while the first message is processed (otherwise SQS makes it visible again and delivers it a second time)]
Expecting value to be true but was false
{noformat}

h3. Proposed fix

{{processBatch}} registers every message of the batch with the extender before it routes the first one, and removes the messages it does not route (the consumer is stopping or suspending, or the processor threw) so that they become visible again as they do today. The extender's own rule (extend an entry when its deadline is near) is unchanged, so a message that is processed quickly is still not extended.

A second new test suspends the route while the first message is processed: the second message is not routed and must not be extended afterwards (it passes on main, where the message is never registered; it guards the removal).

camel-aws2-sqs unit tests with the fix: 77 run, 0 failures, 0 skipped. Not run: the integration tests (LocalStack), a real queue.

How it was found: reading the ack and visibility paths of the SQS consumer in a sweep of acknowledge/commit/delete timing; the test confirmed it.

Affected: main (tested). camel-4.23.x, camel-4.22.x and camel-4.18.x register the message with the extender inside the processing loop the same way (line 151; read, not run).

Duplicate check (2026-10-16): JIRA text "extendMessageVisibility" (CAMEL-10587, CAMEL-18780, CAMEL-17156, CAMEL-5944, CAMEL-16956: none open about the batch), "TimeoutExtender" / "visibility extender" (CAMEL-19562, CAMEL-19656, CAMEL-21550, CAMEL-21659, CAMEL-21769: start delay, batching, deadlines, an error that extended forever; none about the waiting messages of a batch); GitHub pull requests "sqs extend", "Sqs2Consumer visibility" (#9108, #10828, #17262, #16592, #16945, #11102; all merged, none changes where a message is registered).

_Filed with Claude Code on behalf of allthingssecurity._




--
This message was sent by Atlassian Jira
(v8.20.10#820010)