You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by navneet sharma <na...@gmail.com> on 2017/09/27 13:06:12 UTC

Spark job taking 10s to allocate executors and memory before submitting job

Hi,

I am running spark job taking total 18s, in that 8 seconds for actual
processing logic(business logic) and 10s for allocating executors and
memory. How to reduce initial time.

Any ideas how to reduce time before spark job goes to submit state.

thanks,

Navneet Sharma

Re: Spark job taking 10s to allocate executors and memory before submitting job

Posted by Stéphane Verlet <ka...@gmail.com>.

Sounds like such a small job , if you running in on a cluster  have you
consider simply running it locally  (master = local) ?

On Wed, Sep 27, 2017 at 7:06 AM, navneet sharma <navneetsharma0505@gmail.com
> wrote:

> Hi,
>
> I am running spark job taking total 18s, in that 8 seconds for actual
> processing logic(business logic) and 10s for allocating executors and
> memory. How to reduce initial time.
>
> Any ideas how to reduce time before spark job goes to submit state.
>
> thanks,
>
> Navneet Sharma
>