You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Sachit Murarka <co...@gmail.com> on 2019/05/28 08:05:28 UTC

1 task per executor

Hi All,

I am using spark 2.2
I have enabled spark dynamic allocation with executor cores 4, driver cores
4 and executor memory 12GB driver memory 10GB.

In Spark UI, I see only 1 task is launched per executor.

Could anyone please help on this?

Kind Regards,
Sachit Murarka

Re: 1 task per executor

Posted by Arnaud LARROQUE <al...@gmail.com>.

Hi,

How many files do you read ? Are they splittable ?
If you have 4 files non splittable, your dataset would have 4 partitions
and you will only see one task per partition handle by on executor

Regards,
Arnaud

On Tue, May 28, 2019 at 10:06 AM Sachit Murarka <co...@gmail.com>
wrote:

> Hi All,
>
> I am using spark 2.2
> I have enabled spark dynamic allocation with executor cores 4, driver
> cores 4 and executor memory 12GB driver memory 10GB.
>
> In Spark UI, I see only 1 task is launched per executor.
>
> Could anyone please help on this?
>
> Kind Regards,
> Sachit Murarka
>