You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Shiyao Ma <i...@introo.me> on 2015/06/18 04:48:19 UTC

Why does driver transfer application jar to executors?

Hi,

Looking from my executor logs, the submitted application jar is
transmitted to each executors?

Why does spark do the above? To my understanding, the tasks to be run
are already serialized with TaskDescription.


Regards.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Why does driver transfer application jar to executors?

Posted by Jeff Zhang <zj...@gmail.com>.
TaskDescription only serialize the jar path not the jar content. Multiple
tasks can run on the same executor. Executor will check whether the jar has
been fetched when each time task is launched. If so, it won't fetch it
again.
Only serialize the jar path can prevent serialize jar multiple times which
is inefficient.

On Thu, Jun 18, 2015 at 10:48 AM, Shiyao Ma <i...@introo.me> wrote:

> Hi,
>
> Looking from my executor logs, the submitted application jar is
> transmitted to each executors?
>
> Why does spark do the above? To my understanding, the tasks to be run
> are already serialized with TaskDescription.
>
>
> Regards.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 
Best Regards

Jeff Zhang

Re: Why does driver transfer application jar to executors?

Posted by Dick Davies <di...@hellooperator.net>.
You'd need the jar file (holding class definitions etc.) to do the
deserialisation on the executor.

On 18 June 2015 at 03:48, Shiyao Ma <i...@introo.me> wrote:
> Hi,
>
> Looking from my executor logs, the submitted application jar is
> transmitted to each executors?
>
> Why does spark do the above? To my understanding, the tasks to be run
> are already serialized with TaskDescription.
>
>
> Regards.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org