You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Shardul Mahadik (Jira)" <ji...@apache.org> on 2021/04/14 15:14:00 UTC
[jira] [Created] (SPARK-35073) SparkContext.addJar with an ivy path
should not fail with a custom ivySettings file in non-YARN cluster modes
Shardul Mahadik created SPARK-35073:
---------------------------------------
Summary: SparkContext.addJar with an ivy path should not fail with a custom ivySettings file in non-YARN cluster modes
Key: SPARK-35073
URL: https://issues.apache.org/jira/browse/SPARK-35073
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.2.0
Reporter: Shardul Mahadik
SPARK-33084 introduced support for Ivy paths in sc.addJar or Spark SQL ADD JAR. If we use a custom ivySettings file using spark.jars.ivySettings, it is loaded at https://github.com/apache/spark/blob/b26e7b510bbaee63c4095ab47e75ff2a70e377d7/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L1280. However, this file is only accessible on the client machine. In cluster mode, this file is not available on the driver and so addJar fails. SPARK-34472 provides repro steps for the same.
In SPARK-34472, we decided to restrict the scope of the work to only fix the issue in the YARN resource manager. This fix should also be made to other resource managers which can run in cluster mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org