You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Moiz S Jinia <mo...@gmail.com> on 2017/05/30 07:49:30 UTC
Kafka partitions -> task slots? (keyed stream)
For a keyed stream (where the key is also the message key in the source
kafka topic), is the parallelism of the job restricted to the number of
partitions in the topic?
Source topic has 5 partitions, but available task slots are 12. (3 task
managers each with 4 slots)
Moiz
Re: Kafka partitions -> task slots? (keyed stream)
Posted by Moiz S Jinia <mo...@gmail.com>.
I have just 1 job (that has a ProcessFunction with timers).
You're saying that giving more task slots to my job than the number of
partitions on the source topic is not going to help.
This implies that 1 partition cannot be assigned to more than 1 task slot.
That makes sense as otherwise ordering for a partition would not be
guaranteed.
Thanks.
On Tue, May 30, 2017 at 8:43 PM, Stefan Richter <s.richter@data-artisans.com
> wrote:
> Hi,
>
> it is not restricting the parallelism of your job. Only increasing the
> parallelism of your Job’s sources to more than 5 will not bring any
> improvements. All other operators could still benefit from a higher
> parallelism.
>
> > Am 30.05.2017 um 09:49 schrieb Moiz S Jinia <mo...@gmail.com>:
> >
> > For a keyed stream (where the key is also the message key in the source
> kafka topic), is the parallelism of the job restricted to the number of
> partitions in the topic?
> >
> > Source topic has 5 partitions, but available task slots are 12. (3 task
> managers each with 4 slots)
> >
> > Moiz
>
>
Re: Kafka partitions -> task slots? (keyed stream)
Posted by Stefan Richter <s....@data-artisans.com>.
Hi,
it is not restricting the parallelism of your job. Only increasing the parallelism of your Job’s sources to more than 5 will not bring any improvements. All other operators could still benefit from a higher parallelism.
> Am 30.05.2017 um 09:49 schrieb Moiz S Jinia <mo...@gmail.com>:
>
> For a keyed stream (where the key is also the message key in the source kafka topic), is the parallelism of the job restricted to the number of partitions in the topic?
>
> Source topic has 5 partitions, but available task slots are 12. (3 task managers each with 4 slots)
>
> Moiz