[GitHub] [spark] shardulm94 edited a comment on pull request #31591: [SPARK-34472][YARN] Ship ivySettings file to driver in cluster mode

[GitBox <gi...@apache.org>]

shardulm94 edited a comment on pull request #31591:
URL: https://github.com/apache/spark/pull/31591#issuecomment-812750599


   @tgravescs @xkrogen `spark.jar.ivySettings` today only supports files present on the client. It does not support `local://` or `hdfs://` files. Can the scope of this PR be restricted to making the current file support work in cluster mode?
   
   Trying to add `local://` and `hdfs://` support would be nice, but there doesn't seem to be a requirement yet and it is making things unnecessarily complicated. I think most of the complexity stems from the fact the ivySettings loading code in SparkSubmit is shared between both the Spark client and the Spark driver, the way to access the files in these two contexts seems very different.
   
   Since the restriction for ivySettings file being client local already existed before, I am guessing most users who need a custom ivySettings file have it configured locally and we can address adding support for other schemes later if need arises.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org