You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by mb...@gmail.com,
mb...@gmail.com on 2019/03/13 08:52:57 UTC
Partitions and the number of cores/executors
Hi,
I am working with Gelly graph library but I think the question is applicable in general. I just want to confirm if a single data partition in Flink is executed by only a single executor/core? i.e. multiple executors can't be utilized to process a single partition in parallel. So, if I need to have utilize higher parallelism I should simply have an equal number of partitions.
I suppose the reason for this is to avoid any kind of locking that might be required to process a data partition with multiple executors.
Thanks
- Bilal
Re: Partitions and the number of cores/executors
Posted by Stefan Richter <s....@ververica.com>.
Hi,
Your assumption is right. Parallel processing is based in splitting inputs and each split is only processed by one task instance at a time.
Best,
Stefan
> On 13. Mar 2019, at 09:52, mbilalce.dev@gmail.com wrote:
>
> Hi,
>
> I am working with Gelly graph library but I think the question is applicable in general. I just want to confirm if a single data partition in Flink is executed by only a single executor/core? i.e. multiple executors can't be utilized to process a single partition in parallel. So, if I need to have utilize higher parallelism I should simply have an equal number of partitions.
>
> I suppose the reason for this is to avoid any kind of locking that might be required to process a data partition with multiple executors.
>
>
> Thanks
>
> - Bilal
>