You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by igyu <ig...@21cn.com> on 2021/08/20 15:49:39 UTC

How can I use sparkContext.addFile

in spark-shell
I can run 

val url = "hdfs://nameservice1/user/jztwk/config.json"
Spark.sparkContext.addFile(url)
val json_str = readLocalFile(SparkFiles.get(url.split("/").last))

but when I make jar package

spark-submit --master yarn --deploy-mode cluster --principal jztwk/hadoop@JOIN.COM --keytab /hadoop/app/jztwk.keytab --class com.join.Synctool --jars hdfs://nameservice1/sparklib/* jztsynctools-1.0-SNAPSHOT.jar

I get a error

 ERROR yarn.Client: Application diagnostics message: User class threw exception: java.io.FileNotFoundException: /hadoop/yarn/nm1/usercache/jztwk/appcache/application_1627287887991_0571/spark-020a769c-6d9c-42ff-9bb2-1407cf6ed0bc/userFiles-1f57a3ed-22fa-4464-84e4-e549685b0d2d/hadoop/yarn/nm1/usercache/jztwk/appcache/application_1627287887991_0571/spark-020a769c-6d9c-42ff-9bb2-1407cf6ed0bc/userFiles-1f57a3ed-22fa-4464-84e4-e549685b0d2d/config.json (No such file or directory)




but 



igyu

Re: How can I use sparkContext.addFile

Posted by Gourav Sengupta <go...@gmail.com>.

Hi,

why are you using add file for a json file? Cant you just read it as a
dataframe?

Regards,
Gourav Sengupta

On Fri, Aug 20, 2021 at 4:50 PM igyu <ig...@21cn.com> wrote:

> in spark-shell
> I can run
>
> val url = "hdfs://nameservice1/user/jztwk/config.json"
> Spark.sparkContext.addFile(url)
> val json_str = readLocalFile(SparkFiles.get(url.split("/").last))
>
> but when I make jar package
>
> spark-submit --master yarn --deploy-mode cluster --principal jztwk/
> hadoop@JOIN.COM --keytab /hadoop/app/jztwk.keytab --class
> com.join.Synctool --jars hdfs://nameservice1/sparklib/*
> jztsynctools-1.0-SNAPSHOT.jar
>
> I get a error
>
>  ERROR yarn.Client: Application diagnostics message: User class threw
> exception: java.io.FileNotFoundException:
> /hadoop/yarn/nm1/usercache/jztwk/appcache/application_1627287887991_0571/spark-020a769c-6d9c-42ff-9bb2-1407cf6ed0bc/userFiles-1f57a3ed-22fa-4464-84e4-e549685b0d2d/hadoop/yarn/nm1/usercache/jztwk/appcache/application_1627287887991_0571/spark-020a769c-6d9c-42ff-9bb2-1407cf6ed0bc/userFiles-1f57a3ed-22fa-4464-84e4-e549685b0d2d/config.json
> (No such file or directory)
>
>
>
>
> but
>
> ------------------------------
> igyu
>