[GitHub] [spark] tgravescs commented on pull request #31591: [SPARK-34472][YARN] Ship ivySettings file to driver in cluster mode

[GitBox <gi...@apache.org>]

tgravescs commented on pull request #31591:
URL: https://github.com/apache/spark/pull/31591#issuecomment-788079584


   
   so I guess this case is just if the user does sc.addJar("ivy:..") so you wouldn't know at submission time and can't necessarily download the file in client before submission. One issue with yarn clusters and likely k8s cluster is that network might be more restricted on the cluster itself rather than the client where you submit.  But I guess either way that would fail.
   
   With k8s if everything isn't in the docker image you would have to configure a separate path to upload things to (like s3). It looks like this change is only for YARN so I'm assuming this doesn't work with k8s either then? Or standalone mode with cluster mode?  I'm fine with fixing for just yarn with this but we should document what it works for.
   
   note that spark specifically supports the "local:" prefix to mean local files.  This means we shouldn't upload them as they are local to each machine. https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L103
   
   Is there a reason we don't just call distributed() like we do for key tab file or the app jar for instance?  If we are worried about it conflicting with something else in --files then just give it a unique name


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org