[jira] [Created] (SPARK-33083) support submission client remote/local dependency check in k8s cluster mode

["Dexter Hu (Jira)" <ji...@apache.org>]

Dexter Hu created SPARK-33083:
---------------------------------

             Summary: support submission client remote/local dependency check in k8s cluster mode
                 Key: SPARK-33083
                 URL: https://issues.apache.org/jira/browse/SPARK-33083
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes, Spark Submit
    Affects Versions: 2.4.7
            Reporter: Dexter Hu
             Fix For: 2.4.7


Usecase:
 # Users use Apache Livy to submit spark jobs.  Livy pod's spark-submit command will be invoked to submit jobs to k8e cluster with *--deploy-mode cluster*
 # The Livy pod and future driver/executor pod have *different* permissions to access remote dependencies specified by --files, --jars, --py-files from S3, gcs, hdfs://

Since the submission client host or Livy pod in this case don't have permissions to download remote files, it is possible to support options to *disable* remote resource resolution/download at Livy pod or submission client host? 

Of course, users will make sure these remote files are accessible by driver and executor pods with secure tokens, but not for Livy pod.

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org