You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by SURAJ KADEL <su...@gmail.com> on 2024/12/09 15:34:15 UTC

Flink 1.16.3 parallelism issue

Hi Team,

I am currently running Flink 1.16.3 with Apache Beam to run our batch jobs.
We are trying to process the data with an input size of over 20 million,
but unfortunately, it is failing.
we want Flink to handle the parallelism based on input load automatically
but even with -1 configuration, we have observed that the parallelism was
set to 1 and SDK workers were not automatically scaling.
we tried all sorts of configurations and their combinations but could not
get the desired result.

Is there some bug concerning parallelism in the 1.16.3 version?

Thanks,
Suraj