Re: Spark Yarn Cluster with Reference File

[Aditya <ad...@augmentiq.co.in>]

Hi Abhishek,

 From your spark-submit it seems your passing the file as a parameter to 
the driver program. So now it depends what exactly you are doing with 
that parameter. Using --files option it will be available to all the 
worker nodes but if in your code if you are referencing using the 
specified path in distributed mode it wont get the file on the worker nodes.

If you can share the snippet of code it will be easy to debug.

On Friday 23 September 2016 01:03 PM, ABHISHEK wrote:
> Hello there,
>
> I have Spark Application which refer to an external file \u2018abc.drl\u2019 and 
> having unstructured data.
> Application is able to find this reference file if I  run app in Local 
> mode but in Yarn with Cluster mode, it is not able to  find the file 
> in the specified path.
> I tried with both local and hdfs path with \u2013-files option but it 
> didn\u2019t work.
>
>
> What is working ?
> 1.Current  Spark Application runs fine if I run it in Local mode as 
> mentioned below.
> In below command   file path is local path not HDFS.
> spark-submit --master local[*]  --class "com.abc.StartMain" 
> abc-0.0.1-SNAPSHOT-jar-with-dependencies.jar /home/abhietc/abc/abc.drl
>
> 3.I want to run this Spark application using Yarn with cluster mode.
> For that, I used below command but application is not able to find the 
> path for the reference file abc.drl.I tried giving both local and HDFS 
> path but didn\u2019t work.
>
> spark-submit --master yarn --deploy-mode cluster  --files 
> /home/abhietc/abc/abc.drl --class com.abc.StartMain 
> abc-0.0.1-SNAPSHOT-jar-with-dependencies.jar /home/abhietc/abc/abc.drl
>
> spark-submit --master yarn --deploy-mode cluster  --files 
> hdfs://abhietc.com:8020/user/abhietc/abc.drl 
> <http://abhietc.com:8020/user/abhietc/abc.drl> --class 
> com.abc.StartMain abc-0.0.1-SNAPSHOT-jar-with-dependencies.jar 
> hdfs://abhietc.com:8020/user/abhietc/abc.drl 
> <http://abhietc.com:8020/user/abhietc/abc.drl>
>
> spark-submit --master yarn --deploy-mode cluster  --files 
> hdfs://abc.com:8020/tmp/abc.drl <http://abc.com:8020/tmp/abc.drl> 
> --class com.abc.StartMain abc-0.0.1-SNAPSHOT-jar-with-dependencies.jar 
> hdfs://abc.com:8020/tmp/abc.drl <http://abc.com:8020/tmp/abc.drl>
>
>
> Error Messages:
> Surprising we are not doing any Write operation on reference file but 
> still log shows that application is trying to write to file instead 
> reading the file.
> Also log shows File not found exception for both HDFS and Local path.
> -------------
> 16/09/20 14:49:50 ERROR scheduler.JobScheduler: Error running job 
> streaming job 1474363176000 ms.0
> org.apache.spark.SparkException: Job aborted due to stage failure: 
> Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 
> in stage 1.0 (TID 4, abc.com <http://abc.com>): 
> java.lang.RuntimeException: Unable to write Resource: 
> FileResource[file=hdfs:/abc.com:8020/user/abhietc/abc.drl 
> <http://abc.com:8020/user/abhietc/abc.drl>]
>         at 
> org.drools.compiler.kie.builder.impl.KieFileSystemImpl.write(KieFileSystemImpl.java:71)
>         at 
> com.hmrc.taxcalculator.KieSessionFactory$.getNewSession(KieSessionFactory.scala:49)
>         at 
> com.hmrc.taxcalculator.KieSessionFactory$.getKieSession(KieSessionFactory.scala:21)
>         at 
> com.hmrc.taxcalculator.KieSessionFactory$.execute(KieSessionFactory.scala:27)
>         at 
> com.abc.StartMain$$anonfun$main$1$$anonfun$4.apply(TaxCalculatorMain.scala:124)
>         at 
> com.abc.StartMain$$anonfun$main$1$$anonfun$4.apply(TaxCalculatorMain.scala:124)
>         at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>         at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
>         at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at 
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>         at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
>         at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
>         at 
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
>         at org.apache.spark.scheduler.Task.run(Task.scala:89)
>         at 
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
>         at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.FileNotFoundException: 
> hdfs:/abc.com:8020/user/abhietc/abc.drl 
> <http://abc.com:8020/user/abhietc/abc.drl> (No such file or directory)
>         at java.io.FileInputStream.open(Native Method)
>         at java.io.FileInputStream.<init>(FileInputStream.java:146)
>         at 
> org.drools.core.io.impl.FileSystemResource.getInputStream(FileSystemResource.java:123)
>         at 
> org.drools.compiler.kie.builder.impl.KieFileSystemImpl.write(KieFileSystemImpl.java:58)
>         ... 19 more
> --------------
> Cheers,
> Abhishek
>