You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Madan D via user <us...@flink.apache.org> on 2023/05/09 17:57:54 UTC

Flink Kafka Source rebalancing - 1.14.2

Hello Team,
 We have been using Flink Kafka consumer and recently we have been moving to Flink Kafka source to get more advanced features but we have been observing more rebalances right after data consumed and moving to next operator than Flink Kafka consumer.

Can you please let us know what might be causing even though we are using same parallelism in old and new jobs and we are on Flink 1.14.2.

Regards,Madan 

Re: Flink Kafka Source rebalancing - 1.14.2

Posted by Madan D via user <us...@flink.apache.org>.
 Hello Shammon/Team,
We are using same Flink version which is 1.14.2 but only change is earlier we were using Flink Kafka Consumer not we are moving with Kafka Source. I dont see any difference in Job planner but I see Kafka source is introducing more latency while performing Flinka Kafka consumer.

Attached job plans for both.
Regards,Madan
    On Tuesday, 9 May 2023 at 05:50:49 pm GMT-7, Shammon FY <zj...@gmail.com> wrote:  
 
 Hi Madan,
Could you give the old and new versions of flink and provide the job plan? I think it will help community to find the root cause
Best,Shammon FY
On Wed, May 10, 2023 at 2:04 AM Madan D via user <us...@flink.apache.org> wrote:

Hello Team,
 We have been using Flink Kafka consumer and recently we have been moving to Flink Kafka source to get more advanced features but we have been observing more rebalances right after data consumed and moving to next operator than Flink Kafka consumer.

Can you please let us know what might be causing even though we are using same parallelism in old and new jobs and we are on Flink 1.14.2.

Regards,Madan 

Re: Flink Kafka Source rebalancing - 1.14.2

Posted by Shammon FY <zj...@gmail.com>.
Hi Madan,

Could you give the old and new versions of flink and provide the job plan?
I think it will help community to find the root cause

Best,
Shammon FY

On Wed, May 10, 2023 at 2:04 AM Madan D via user <us...@flink.apache.org>
wrote:

> Hello Team,
>
> We have been using Flink Kafka consumer and recently we have been moving
> to Flink Kafka source to get more advanced features but we have been
> observing more rebalances right after data consumed and moving to next
> operator than Flink Kafka consumer.
>
> Can you please let us know what might be causing even though we are using
> same parallelism in old and new jobs and we are on Flink 1.14.2.
>
>
> Regards,
> Madan
>