You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Nathan McCarthy (JIRA)" <ji...@apache.org> on 2015/06/19 06:39:00 UTC

[jira] [Created] (SPARK-8475) SparkSubmit with Ivy jars is very slow to load when no internet access

Nathan McCarthy created SPARK-8475:
--------------------------------------

             Summary: SparkSubmit with Ivy jars is very slow to load when no internet access
                 Key: SPARK-8475
                 URL: https://issues.apache.org/jira/browse/SPARK-8475
             Project: Spark
          Issue Type: Bug
          Components: Spark Submit
    Affects Versions: 1.4.0
            Reporter: Nathan McCarthy
            Priority: Minor


Spark Submit adds maven central & spark bintray to the ChainResolver before it adds any external resolvers. 

When running on a cluster without internet access, this means the spark shell takes forever to launch as it tries these two remote repos before the ones specified in the --repositories list.

This is also a problem for users who proxy maven/ivy repos with something like Nexus/Artifactory. 

I see two options for a fix;

* Change the order repos are added to the ChainResolver, making the --repositories supplied repos come before anything else. 
* Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default true) which when false wont add the maven central & bintry to the ChainResolver. 

Happy to do a PR for this fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org