How to avoid spark executor fetching jars in Local mode

[Bin Chen <bc...@talend.com.INVALID>]

Hello everyone,

For spark local mode, I noticed the executor fetching jars from local, is there a way to avoid these steps, as these jars already on the local filesystem, it takes seconds to fetching all the jars, we want to reduce the time on bootstrap phase.

we are using spark-submit with `--conf spark.jars` to define these libs.

[2019-09-26 06:19:59,651] [INFO ] o.a.s.e.Executor - Fetching spark://10.2.1.5:43745/jars/xxx.jar with timestamp 1569478797007
[2019-09-26 06:19:59,729] [INFO ] o.a.s.n.c.TransportClientFactory - Successfully created connection to /10.2.1.5:43745 after 55 ms (0 ms spent in bootstraps)
[2019-09-26 06:19:59,754] [INFO ] o.a.s.u.Utils - Fetching spark://10.2.1.5:43745/jars/xxx.jar to /tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/fetchFileTemp6346264674899227929.tmp
[2019-09-26 06:19:59,801] [INFO ] o.a.s.e.Executor - Adding file:/tmp/spark-0365e48c-1747-4370-978f-7cd142ef0375/userFiles-3309dc5e-b6d0-4b76-a9aa-8e0a226ddab9/xxx.jar to class loader

Thanks

Chen Bin



As a recipient of an email from Talend, your contact personal data will be on our systems. Please see our contacts privacy notice at Talend, Inc. <https://www.talend.com/contacts-privacy-policy/>