[GitHub] [spark] steveloughran commented on issue #23938: [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency, not package it

[GitBox <gi...@apache.org>]

steveloughran commented on issue #23938: [MINOR][DOCS] Clarify that Spark apps should mark Spark as a 'provided' dependency,  not package it
URL: https://github.com/apache/spark/pull/23938#issuecomment-469236386
 
 
   you should compile with hadoop-cloud and add those JARs it pulls in to the spark tarball placed on the shared cluster FS for YARN to pick up. Don't know about other deployment engines I'm afraid. The build also adds it to the SPARK_HOME/lib, which gives it to you for spark-standalone during spark submit, either for anything related to JAR upload, or for any store which implements delegation tokens (HADOOP-14456, HADOOP-16068, etc), so it collects the tokens for all stores listed in spark.yarn.hadoopFilesystems.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org