You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by rahul c <rc...@gmail.com> on 2020/08/01 13:32:23 UTC
Spark-submit --files option help
Hi all,
I am trying to pass some configuration files via spark-submit command in
cluster mode.
From logs I can see the files are transferred to each executors.
But how to build the absolute path of the file in the code?
Can anyone plz guide on it with some references.
Appreciate your help on this.
Thanks and regards
Rahul
Re: Spark-submit --files option help
Posted by Russell Spitzer <ru...@gmail.com>.
You can use SparkFiles.get(path)
Example here
https://github.com/datastax/spark-cassandra-connector/blob/master/connector/src/main/scala/com/datastax/spark/connector/cql/CassandraConnectionFactory.scala#L152
Also this is probably a better question for the user list than the dev one
On Sat, Aug 1, 2020, 8:34 AM rahul c <rc...@gmail.com> wrote:
> Hi all,
>
> I am trying to pass some configuration files via spark-submit command in
> cluster mode.
> From logs I can see the files are transferred to each executors.
> But how to build the absolute path of the file in the code?
>
> Can anyone plz guide on it with some references.
>
> Appreciate your help on this.
>
> Thanks and regards
> Rahul
>
>
>
>