You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by guxiaobo1982 <gu...@qq.com> on 2015/01/18 03:08:51 UTC

How to get the master URL at runtime inside driver program?

Hi,


Driver programs submitted by the spark-submit script will get the runtime spark master URL, but how it get the URL inside the main method when creating the SparkConf object?


Regards,

Re: How to get the master URL at runtime inside driver program?

Posted by Raghavendra Pandey <ra...@gmail.com>.

If you pass spark master URL to spark-submit, you don't need to pass the
same to SparkConf object. You can create SparkConf without this property or
for that matter any other property that you pass in spark-submit.

On Sun, Jan 18, 2015 at 7:38 AM, guxiaobo1982 <gu...@qq.com> wrote:

> Hi,
>
> Driver programs submitted by the spark-submit script will get the runtime
> spark master URL, but how it get the URL inside the main method when
> creating the SparkConf object?
>
> Regards,
>
>

Re: How to get the master URL at runtime inside driver program?

Posted by Tobias Pfeiffer <tg...@preferred.jp>.

Hi,

On Sun, Jan 18, 2015 at 11:08 AM, guxiaobo1982 <gu...@qq.com> wrote:
>
> Driver programs submitted by the spark-submit script will get the runtime
> spark master URL, but how it get the URL inside the main method when
> creating the SparkConf object?
>

The master will be stored in the spark.master property. I use the following
snippet:

    // When run through spark-submit, the Java system property
"spark.master"
    // will contain the master passed to spark-submit and we *must* use the
    // same; otherwise use "local[3]".
    val master = scala.util.Properties.propOrElse("spark.master",
"local[3]")

Tobias