Submit Spark driver on Yarn Cluster in client mode

[Naveen Kumar Pokala <np...@spcapitaliq.com>]

Hi,

I want to submit my spark program from my machine on a YARN Cluster in yarn client mode.

How to specify al l the required details through SPARK submitter.

Please provide me some details.

-Naveen.

Re: Submit Spark driver on Yarn Cluster in client mode

[Akhil Das <ak...@sigmoidanalytics.com>]

You can export the hadoop configurations dir (export HADOOP_CONF_DIR=XXX) in
the environment and then submit it like:

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn-cluster \  # can also be `yarn-client` for client mode
  --executor-memory 20G \
  --num-executors 50 \
  /path/to/examples.jar \
  1000

More details over here
https://spark.apache.org/docs/1.1.0/submitting-applications.html#launching-applications-with-spark-submit

Thanks
Best Regards

On Mon, Nov 24, 2014 at 4:01 PM, Naveen Kumar Pokala <
npokala@spcapitaliq.com> wrote:

> Hi,
>
>
>
> I want to submit my spark program from my machine on a YARN Cluster in
> yarn client mode.
>
>
>
> How to specify al l the required details through SPARK submitter.
>
>
>
> Please provide me some details.
>
>
>
> -Naveen.
>