Flink kafka producer partitioning scheme

[Vishwas Siravara <vs...@gmail.com>]

Hi guys,
From the flink doc
*By default, if a custom partitioner is not specified for the Flink Kafka
Producer, the producer will use a FlinkFixedPartitioner that maps each
Flink Kafka Producer parallel subtask to a single Kafka partition (i.e.,
all records received by a sink subtask will end up in the same Kafka
partition).*

Does this mean that if my downstream topic has 40 partitions , I will need
40 parallel subtasks ?

Thanks,
Vishwas

Re: Flink kafka producer partitioning scheme

[Eduardo Winpenny Tejedor <ed...@gmail.com>]

Hi Vishwas,

Just because a Kafka topic has 40 partitions doesn't mean the 40 partitions
need to have messages in them. However, while it is not a requirement it is
certainly convenient to populate the 40 partitions for load balancing.

What this means if you want to use all partitions is you either need 40
subtasks (as you've said) OR you can override the partitioner to decide to
which partition each message should go to.

Regards,
Eduardo

On Fri, 13 Sep 2019, 17:29 Vishwas Siravara, <vs...@gmail.com> wrote:

> Hi guys,
> From the flink doc
> *By default, if a custom partitioner is not specified for the Flink Kafka
> Producer, the producer will use a FlinkFixedPartitioner that maps each
> Flink Kafka Producer parallel subtask to a single Kafka partition (i.e.,
> all records received by a sink subtask will end up in the same Kafka
> partition).*
>
> Does this mean that if my downstream topic has 40 partitions , I will need
> 40 parallel subtasks ?
>
> Thanks,
> Vishwas
>