Spark Action - Loading additional files with submission

[Daniel Haviv <da...@veracity-group.com>]

Hi,

I have a spark-submit command which uses --files to ship some files to each
worker upon submission.

It is unclear to me how can I achieve this using the Spark Action.

I’ve tried using oozie.libpath in hopes that it will copy all the files
from the hdfs location I set to the local dir and I could just add the
--files argument but that didn’t work.



Here is my workflow.xml:



<workflow-app name="GenericAlgo_spark" xmlns="uri:oozie:workflow:0.5">

    <start to="spark-c6ae"/>

    <kill name="Kill">

        <message>Action failed, error
message[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <action name="spark-c6ae">

        <spark xmlns="uri:oozie:spark-action:0.1">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <master>yarn</master>

            <mode>cluster</mode>

            <name>Daniel</name>


<class>com.mycompany.algorith_wrapper.Generic_Algo.GenericAlgoDriver</class>


<jar>hdfs:///mycompany/workflows/GenericAlgo/Generic-Algo-v1.0-SNAPSHOT-jar-with-dependencies.jar</jar>

              <arg>--num-executors</arg>

              <arg>4</arg>

              <arg>--driver-java-options</arg>

              <arg>-Dlog4j.configuration=log4j.properties</arg>

              <arg>--conf</arg>


<arg>spark.executor.extraJavaOptions=&quot;-Dlog4j.configuration=log4j.properties&quot;</arg>

              <arg>--driver-memory</arg>

              <arg>4096m</arg>

              <arg>--files</arg>


          <arg>DefaultSettings_1440110506.net,frSignatureGenerator.lua,lbpcascade_frontalface.xml,log4j.properties,config.properties</arg>

              <arg>--executor-memory</arg>

              <arg>4096m</arg>

              <arg>--executor-cores</arg>

              <arg>2</arg>

        </spark>

        <ok to="End"/>

        <error to="Kill"/>

    </action>

    <end name="End"/>

</workflow-app>



Any ideas?





Thank you,

Daniel.