[jira] [Commented] (SPARK-8475) SparkSubmit with Ivy jars is very slow to load with no internet access

["Sean Owen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-8475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14593158#comment-14593158 ] 

Sean Owen commented on SPARK-8475:
----------------------------------

For what it's worth, I like option 1.

> SparkSubmit with Ivy jars is very slow to load with no internet access
> ----------------------------------------------------------------------
>
>                 Key: SPARK-8475
>                 URL: https://issues.apache.org/jira/browse/SPARK-8475
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Submit
>    Affects Versions: 1.4.0
>            Reporter: Nathan McCarthy
>            Priority: Minor
>
> Spark Submit adds maven central & spark bintray to the ChainResolver before it adds any external resolvers. 
> https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L821
> When running on a cluster without internet access, this means the spark shell takes forever to launch as it tries these two remote repos before the ones specified in the --repositories list. In our case we have a proxy which the cluster can access it and supply it via --repositories.
> This is also a problem for users who maintain a proxy for maven/ivy repos with something like Nexus/Artifactory. Having a repo proxy is popular at many organisations so I'd say this would be a useful change for these users as well. In the current state even if a maven central proxy is supplied, it will still try and hit central. 
> I see two options for a fix;
> * Change the order repos are added to the ChainResolver, making the --repositories supplied repos come before anything else. https://github.com/apache/spark/blob/branch-1.4/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L843 
> * Have a config option (like spark.jars.ivy.useDefaultRemoteRepos, default true) which when false wont add the maven central & bintry to the ChainResolver. 
> Happy to do a PR for this fix. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org