flink pipeline canary upgrade question

[Sigalit Eliazov <e....@gmail.com>]

Hi,

We are exploring options to support canary upgrades for our Flink pipelines
and have considered the following approach.

   1.

   *Stateless Pipelines*: For stateless pipelines, we define two clusters,
   job-v1 and job-v2, using the same consumer group to avoid event
   duplication. To control the data handled by each job, we set different
   parallelism levels on the consumer. This approach works well as long as no
   state is involved.
   2.

   *Stateful Pipelines*: In the case of stateful pipelines, we need to
   ensure that each cluster receives events that are properly partitioned by
   key. The solution we've found so far is to explicitly assign partitions per
   cluster in the consumer definition for each job. Is there an alternative
   method to manage this, ideally through configuration rather than explicit
   code changes?

We would appreciate any insights or suggestions

BTW, we are using flink 1.18 and flink operator 1.7.

Thanks

Sigalit