You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Tomer Benyamini <to...@gmail.com> on 2014/12/11 19:27:40 UTC

custom spark app name in yarn-cluster mode

Hi,

I'm trying to set a custom spark app name when running a java spark app in
yarn-cluster mode.

 SparkConf sparkConf = new SparkConf();

 sparkConf.setMaster(System.getProperty("spark.master"));

 sparkConf.setAppName("myCustomName");

 sparkConf.set("spark.logConf", "true");

 JavaSparkContext sc = new JavaSparkContext(sparkConf);


Apparently this only works when running in yarn-client mode; in
yarn-cluster mode the app name is the class name, when viewing the app in
the cluster manager UI. Any idea?


Thanks,

Tomer

Re: custom spark app name in yarn-cluster mode

Posted by Tomer Benyamini <to...@gmail.com>.

Thanks Sandy, passing --name works fine :)

Tomer

On Fri, Dec 12, 2014 at 9:35 AM, Sandy Ryza <sa...@cloudera.com> wrote:
>
> Hi Tomer,
>
> In yarn-cluster mode, the application has already been submitted to YARN
> by the time the SparkContext is created, so it's too late to set the app
> name there.  I believe giving it with the --name property to spark-submit
> should work.
>
> -Sandy
>
> On Thu, Dec 11, 2014 at 10:28 AM, Tomer Benyamini <to...@gmail.com>
> wrote:
>>
>>
>>
>> On Thu, Dec 11, 2014 at 8:27 PM, Tomer Benyamini <to...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to set a custom spark app name when running a java spark app
>>> in yarn-cluster mode.
>>>
>>>  SparkConf sparkConf = new SparkConf();
>>>
>>>  sparkConf.setMaster(System.getProperty("spark.master"));
>>>
>>>  sparkConf.setAppName("myCustomName");
>>>
>>>  sparkConf.set("spark.logConf", "true");
>>>
>>>  JavaSparkContext sc = new JavaSparkContext(sparkConf);
>>>
>>>
>>> Apparently this only works when running in yarn-client mode; in
>>> yarn-cluster mode the app name is the class name, when viewing the app in
>>> the cluster manager UI. Any idea?
>>>
>>>
>>> Thanks,
>>>
>>> Tomer
>>>
>>>
>>>
>>

Re: custom spark app name in yarn-cluster mode

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi Tomer,

In yarn-cluster mode, the application has already been submitted to YARN by
the time the SparkContext is created, so it's too late to set the app name
there.  I believe giving it with the --name property to spark-submit should
work.

-Sandy

On Thu, Dec 11, 2014 at 10:28 AM, Tomer Benyamini <to...@gmail.com>
wrote:
>
>
>
> On Thu, Dec 11, 2014 at 8:27 PM, Tomer Benyamini <to...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to set a custom spark app name when running a java spark app
>> in yarn-cluster mode.
>>
>>  SparkConf sparkConf = new SparkConf();
>>
>>  sparkConf.setMaster(System.getProperty("spark.master"));
>>
>>  sparkConf.setAppName("myCustomName");
>>
>>  sparkConf.set("spark.logConf", "true");
>>
>>  JavaSparkContext sc = new JavaSparkContext(sparkConf);
>>
>>
>> Apparently this only works when running in yarn-client mode; in
>> yarn-cluster mode the app name is the class name, when viewing the app in
>> the cluster manager UI. Any idea?
>>
>>
>> Thanks,
>>
>> Tomer
>>
>>
>>
>

Re: custom spark app name in yarn-cluster mode

Posted by Tomer Benyamini <to...@gmail.com>.

On Thu, Dec 11, 2014 at 8:27 PM, Tomer Benyamini <to...@gmail.com>
wrote:

> Hi,
>
> I'm trying to set a custom spark app name when running a java spark app in
> yarn-cluster mode.
>
>  SparkConf sparkConf = new SparkConf();
>
>  sparkConf.setMaster(System.getProperty("spark.master"));
>
>  sparkConf.setAppName("myCustomName");
>
>  sparkConf.set("spark.logConf", "true");
>
>  JavaSparkContext sc = new JavaSparkContext(sparkConf);
>
>
> Apparently this only works when running in yarn-client mode; in
> yarn-cluster mode the app name is the class name, when viewing the app in
> the cluster manager UI. Any idea?
>
>
> Thanks,
>
> Tomer
>
>
>