[jira] [Resolved] (CAMEL-21659) Improve aws sqs TimeoutExtender by introducing deadline concept to messages

["Claus Ibsen (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-21659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-21659.
---------------------------------
    Resolution: Fixed

Thanks

> Improve aws sqs TimeoutExtender by introducing deadline concept to messages
> ---------------------------------------------------------------------------
>
>                 Key: CAMEL-21659
>                 URL: https://issues.apache.org/jira/browse/CAMEL-21659
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws2-sqs
>    Affects Versions: 4.8.3
>            Reporter: Simon Rasmussen
>            Assignee: Simon Rasmussen
>            Priority: Major
>             Fix For: 4.8.4, 4.10.0
>
>
> After upgrading to 4.8.3, we started seeing the following warnings in our logs, quite often:
> {noformat}
> 2025-01-27T06:54:34.226Z
> WARN 11 --- [X] [TimeoutExtender]
> o.a.c.component.aws2.sqs.Sqs2Consumer ::
> [BatchResultErrorEntry(Id=FBDB3014CD64A96-0000000000093816,
> SenderFault=true, Code=ReceiptHandleIsInvalid, Message=Message does
> not exist or is not available for visibility timeout change)]{noformat}
> The error occurs when an attempt to extend message visibility for a message that has already been deleted in SQS.
> In CAMEL-19656, the implementation was switched to using the batched call, but failed to implement the proper ignore in case of above error – as the old implementation had.
> I also noticed that the current implementation extends ALL messages in the queue of non-processed – not just those that are about to expire. This happens because the TimeoutExtender, currently will make a copy of all the values in the Map, and extend them, in batches.
> In case of 100 messages, this will require 10 requests, even though the last messages in the queue might not need to be extended yet.
> If the route is used together with the threads EIP, then the default queue size is 1000, which makes above algorithm really inefficient.
> I suggest introducing a concept of a deadline to each entry in the TimeoutExtender. The deadline is set to now() plus 80% of the visibilityTimeout. When sending requests, only messages that will expire before the next execution of the TimeoutExtender will be extended. The TimeoutExtender task itself should run more frequently, but will pick up less messages each time. I suggest it running every visibilityTimeout/2. In a typical configuration, this would result in executing TimeoutExtender every 10 seconds.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)