[jira] [Resolved] (CAMEL-8221) No backpressure in aws-sqs Consumer

["Claus Ibsen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CAMEL-8221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Claus Ibsen resolved CAMEL-8221.
--------------------------------
    Resolution: Invalid

More a question for the user forum / mailing list

> No backpressure in aws-sqs Consumer
> -----------------------------------
>
>                 Key: CAMEL-8221
>                 URL: https://issues.apache.org/jira/browse/CAMEL-8221
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws
>    Affects Versions: 2.14.1
>            Reporter: Dan Brown
>             Fix For: Future
>
>
> We're using a camel Consumer with an aws-sqs endpoint and running into issues with large queues and slow consumers. The consumer jvm downloads messages very fast from sqs, onto the jvm heap, even though the camel receive method processes each message very slowly (and Ack's when done, with autoAck = false). The result of this is that the consumer jvm continually fills its heap with an unbounded queue of incoming messages and eventually throws OOME.
> To avoid this failure, we're looking for a way to enable backpressure when using aws-sqs—e.g. an on-heap bounded queue between the component fetching from sqs and the user-defined actor processing the messages—but I don't see anything relevant in the config:
> - http://camel.apache.org/aws-sqs.html
> Looking at the code, I see that SqsConsumer subtypes ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get backpressure, should it use something like EventDrivenPollingConsumer somewhere, which uses a blocking queue to avoid the heap blowup above?
> - https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
> - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)