Backpressure with aws-sqs Consumer?

[Dan Brown <jd...@gmail.com>]

Hello,

We're using a camel Consumer with an aws-sqs endpoint and running into
issues with large queues and slow consumers. The consumer jvm appears to
download messages very fast from sqs, onto the jvm heap, even though the
camel receive method processes each message very slowly (and Ack's when
done, with autoAck = false). The result of this is that the consumer jvm
continually fills its heap with an unbounded queue of incoming messages and
eventually throws OOME.

To avoid this failure, is there a way to enable backpressure when using
aws-sqs?—e.g. an on-heap bounded queue between the component fetching from
sqs and the user-defined actor processing the messages. I didn't see
anything relevant in the config:
- http://camel.apache.org/aws-sqs.html

And I didn't see any relevant jira issues:
-
https://issues.apache.org/jira/browse/CAMEL-5113?jql=project%20%3D%20CAMEL%20AND%20text%20~%20sqs

Looking at the code, I see that SqsConsumer subtypes
ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get
backpressure, should it use something like EventDrivenPollingConsumer
somewhere, which uses a blocking queue to avoid the heap blowup above?
-
https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43
-
https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42

Thanks,
Dan



--
View this message in context: http://camel.465427.n5.nabble.com/Backpressure-with-aws-sqs-Consumer-tp5761348.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Backpressure with aws-sqs Consumer?

[Dan Brown <jd...@gmail.com>]

I went ahead and opened this as a bug:
https://issues.apache.org/jira/browse/CAMEL-8221



--
View this message in context: http://camel.465427.n5.nabble.com/Backpressure-with-aws-sqs-Consumer-tp5761348p5761478.html
Sent from the Camel - Users mailing list archive at Nabble.com.