[GitHub] [spark] shardulm94 commented on a change in pull request #31591: [SPARK-34472][CORE] Ship ivySettings file to driver in cluster mode

[GitBox <gi...@apache.org>]

shardulm94 commented on a change in pull request #31591:
URL: https://github.com/apache/spark/pull/31591#discussion_r580946284



##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -1278,11 +1287,22 @@ private[spark] object SparkSubmitUtils {
       remoteRepos: Option[String],
       ivyPath: Option[String]): IvySettings = {
     val file = new File(settingsFile)
-    require(file.exists(), s"Ivy settings file $file does not exist")
-    require(file.isFile(), s"Ivy settings file $file is not a normal file")

Review comment:
       Yeah, as you pointed out it only works if I have the copy in the current working directory. However we cannot which directory our users launch spark-submit from.
   
   What do you think of this? In Yarn Client, we can add the ivySettings file to `spark.yarn.dist.files` or `__spark__conf__.zip` and then we can modify the property `spark.jars.ivySettings` to change it to `./ivysettings.xml` or `__spark__conf__/ivysettings.xml`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org