[jira] [Commented] (CAMEL-5113) Parallel and fault tolerant message processing for SQS endpoints.

["Christian Posta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14247265#comment-14247265 ] 

Christian Posta commented on CAMEL-5113:
----------------------------------------

Patch here:
https://github.com/christian-posta/camel/commit/43ad71e254107c8d9783018826936fe1050fce90

Would be great if someone on core camel team could review it ([~davsclaus]), would be even better if someone could try it out. What this does is allow you to set a "concurrentConsumers" property that allows you to run multiple threads of the aws-sqs consumers. 

We have deployed this patch in a top e-retailer and processed many many millions of transactions during cyber Monday and it performed with ZERO flaws, while draining a very deep queue very fast... I know, vague, but cannot post the numbers :)

Please give me feedback before I commit it.

> Parallel and fault tolerant message processing for SQS endpoints.
> -----------------------------------------------------------------
>
>                 Key: CAMEL-5113
>                 URL: https://issues.apache.org/jira/browse/CAMEL-5113
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws
>            Reporter: Daniel Carleton
>            Assignee: Christian Posta
>             Fix For: 2.15.0
>
>
> I'm using Camel to implement parallel processing of jobs in an SQS queue, and ran into a few issues with the current implementation of the SQS component:
> # SqsConsumer uses a blocking/synchronous processor, which prevents parallel processing of multiple messages from a single endpoint.
> # Having a maxMessagesPerPoll other than one doesn't seem to make sense, because any messages not being actively processed should be left back in the queue for other consumers to have a chance with.
> # Rollback processing doesn't go back to SQS and set the visibility timeout to zero, which prevents immediate retries. 
> I propose the following solutions to these problems:
> # Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().  (Messages in SQS aren't guaranteed to be FIFO anyway, so there should be no issue with order of processing.)
> # Replace maxMessagesPerPoll with maxInFlightMessages.  Put a semaphore in SqsConsumer to control the maximum number of in flight messages, and when polling SQS always set the number of available permits as the maximum number of messages to retrieve.
> # In the onFailure callback for an exchange set the visibility timeout in SQS to zero via ChangeMessageVisibility.
> How does this sound?  I'm working on a patch.  This is my first work on Camel, so if you see any problems with my approach let me know!
> Thanks,
> - Dan



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)