Flink RabbitMQ connector incompatible with Quorom Queues

[mejri houssem <me...@gmail.com>]

Hello everyone,

So, to explain the context, we are a team who uses flink (V1.16.2) with its
RabbitMQ connector (v1.16.2). For flink we use a PrefetchCount to ensure
that no timeouts are caused for our checkpoints which means we use General
QOS that is hardcoded in the library (I will attach below the snippet of
the function that does so). But now we are migrating to Qurom Queues in
RabbitMQ and that is not compatible with General QOS.
Is there a solution to this? Is there a way to have the prefetchCount
without the General QOS being enabled. Would it be a good idea to have this
field as something that can be personalized rather than hard set in the
library code?


[image: image.png]


Thank you in advance for your help.



Kindest regads,

Re: Flink RabbitMQ connector incompatible with Quorom Queues

[Ahmed Hamdy <ha...@gmail.com>]

Hello Mejri
yes you are correct, the current source implementation would fail to
consume from quorum queues[1]. This should be addressed in next releases of
the connector as I created a ticket for it[2]. Unfortunately meanwhile you
can extend the source yourself to override the configuration.

1- https://www.rabbitmq.com/docs/quorum-queues#global-qos
2- https://issues.apache.org/jira/browse/FLINK-35320
Best Regards
Ahmed Hamdy


On Wed, 8 May 2024 at 14:59, mejri houssem <me...@gmail.com> wrote:

> Hello everyone,
>
> So, to explain the context, we are a team who uses flink (V1.16.2) with
> its RabbitMQ connector (v1.16.2). For flink we use a PrefetchCount to
> ensure that no timeouts are caused for our checkpoints which means we use
> General QOS that is hardcoded in the library (I will attach below the
> snippet of the function that does so). But now we are migrating to Qurom
> Queues in RabbitMQ and that is not compatible with General QOS.
> Is there a solution to this? Is there a way to have the prefetchCount
> without the General QOS being enabled. Would it be a good idea to have this
> field as something that can be personalized rather than hard set in the
> library code?
>
>
> [image: image.png]
>
>
> Thank you in advance for your help.
>
>
>
> Kindest regads,
>