You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Dawid Wysakowicz (Jira)" <ji...@apache.org> on 2020/07/29 18:25:00 UTC
[jira] [Closed] (FLINK-10195) RabbitMQ Source With Checkpointing
Doesn't Backpressure Correctly
[ https://issues.apache.org/jira/browse/FLINK-10195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dawid Wysakowicz closed FLINK-10195.
------------------------------------
Fix Version/s: 1.12.0
Resolution: Fixed
Implemented in 3b6ca3c512f2742c84b3c623ac79c61a04af9ab1
> RabbitMQ Source With Checkpointing Doesn't Backpressure Correctly
> -----------------------------------------------------------------
>
> Key: FLINK-10195
> URL: https://issues.apache.org/jira/browse/FLINK-10195
> Project: Flink
> Issue Type: Bug
> Components: Connectors/ RabbitMQ
> Affects Versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0
> Reporter: Luka Jurukovski
> Assignee: Austin Cawley-Edwards
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The connection between the RabbitMQ server and the client does not appropriately back pressure when auto acking is disabled. This becomes very problematic when a downstream process throttles the data processing to slower then RabbitMQ sends the data to the client.
> The difference in records ends up being stored in the flink's heap space, which grows indefinitely (or technically to "Integer Max" Deliveries). Looking at RabbitMQ's metrics the number of unacked messages looks like steadily rising saw tooth shape.
> Upon further invesitgation it looks like this is due to how the QueueingConsumer works, messages are added to the BlockingQueue faster then they are being removed and processed, resulting in the previously described behavior.
> This may be intended behavior, however this isn't explicitly obvious in the documentation or any of the examples I have seen.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)