You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "jingxiong zhong (Jira)" <ji...@apache.org> on 2022/01/13 07:01:00 UTC
[jira] [Comment Edited] (SPARK-35715) Option "--files" with local:// prefix is not honoured for Spark on kubernetes
[ https://issues.apache.org/jira/browse/SPARK-35715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17475135#comment-17475135 ]
jingxiong zhong edited comment on SPARK-35715 at 1/13/22, 7:00 AM:
-------------------------------------------------------------------
It seems that spark 3 does not support the schema using local as the path. You can try file:///etc/xattr.conf
was (Author: JIRAUSER281124):
t seems that spark 3 does not support the schema using local as the path. You can try file:///etc/xattr.conf
> Option "--files" with local:// prefix is not honoured for Spark on kubernetes
> -----------------------------------------------------------------------------
>
> Key: SPARK-35715
> URL: https://issues.apache.org/jira/browse/SPARK-35715
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 3.0.2, 3.1.2
> Reporter: Pardhu Madipalli
> Priority: Major
>
> When we provide a local file as a dependency using "--files" option, the file is not getting copied to work directories of executors.
> h5. Example 1:
>
> {code:java}
> $SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \
> --deploy-mode cluster \
> --name spark-pi \
> --class org.apache.spark.examples.SparkPi \
> --conf spark.executor.instances=1 \
> --conf spark.kubernetes.container.image=<spark-3.1.2-image> \
> --conf spark.kubernetes.driver.pod.name=sparkdriverpod \
> --files local:///etc/xattr.conf \
> local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000
> {code}
>
> h6. Content of Spark Executor work-dir:
>
> {code:java}
> ~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls /opt/spark/work-dir/
> spark-examples_2.12-3.1.2.jar
> {code}
>
> We can notice here that the file _/etc/xattr.conf_ is *NOT* copied to _/opt/spark/work-dir/ ._
>
> ----
>
> {{Instead of using "–files", if we use "--jars" option the file is getting copied as expected.}}
> h5. Example 2:
> {code:java}
> $SPARK_HOME/bin/spark-submit --master k8s://https://<ip-address-k8s> \
> --deploy-mode cluster \
> --name spark-pi \
> --class org.apache.spark.examples.SparkPi \
> --conf spark.executor.instances=1 \
> --conf spark.kubernetes.container.image=<spark-3.1.2-image> \
> --conf spark.kubernetes.driver.pod.name=sparkdriverpod \
> --jars local:///etc/xattr.conf \
> local:///opt/spark/examples/jars/spark-examples_2.12-3.1.2.jar 1000000
> {code}
> h6. Content of Spark Executor work-dir:
>
> {code:java}
> ~$ kubectl exec -n default spark-pi-22de6279f6bec01c-exec-1 ls /opt/spark/work-dir/
> spark-examples_2.12-3.1.2.jar
> xattr.conf
> {code}
> We can notice here that the file _/etc/xattr.conf_ *IS COPIED* to _/opt/spark/work-dir/ ._
>
> I tested this with versions *3.1.2* and *3.0.2*. It is behaving the same way in both cases.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org