You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Debabrata Ghosh <ma...@gmail.com> on 2017/10/10 11:02:05 UTC

Unable to run Spark Jobs in yarn cluster mode

Hi All,
                    I am constantly hitting an error : "ApplicationMaster:
SparkContext did not initialize after waiting for 100 ms" while running my
Spark code in yarn cluster mode.

Here is the command what I am using :* spark-submit --master yarn
--deploy-mode cluster spark_code.py*

                     Please can you help me with a resolution at your
convenience.

Thanks in advance !

Cheers,

Debu

Re: Unable to run Spark Jobs in yarn cluster mode

Posted by ma...@gmail.com.

Thanks Vadim!

Sent from my iPhone

> On 10-Oct-2017, at 11:09 PM, Vadim Semenov <va...@datadoghq.com> wrote:
> 
> Try increasing the `spark.yarn.am.waitTime` parameter, it's by default set to 100ms which might not be enough in certain cases.
> 
>> On Tue, Oct 10, 2017 at 7:02 AM, Debabrata Ghosh <ma...@gmail.com> wrote:
>> Hi All,
>>                     I am constantly hitting an error : "ApplicationMaster: SparkContext did not initialize after waiting for 100 ms" while running my Spark code in yarn cluster mode.
>> 
>> Here is the command what I am using : spark-submit --master yarn --deploy-mode cluster spark_code.py
>> 
>>                      Please can you help me with a resolution at your convenience.
>> 
>> Thanks in advance !
>> 
>> Cheers,
>> 
>> Debu
>> 
>>                    
>

Re: Unable to run Spark Jobs in yarn cluster mode

Posted by Vadim Semenov <va...@datadoghq.com>.

Try increasing the `spark.yarn.am.waitTime` parameter, it's by default set
to 100ms which might not be enough in certain cases.

On Tue, Oct 10, 2017 at 7:02 AM, Debabrata Ghosh <ma...@gmail.com>
wrote:

> Hi All,
>                     I am constantly hitting an error : "ApplicationMaster:
> SparkContext did not initialize after waiting for 100 ms" while running my
> Spark code in yarn cluster mode.
>
> Here is the command what I am using :* spark-submit --master yarn
> --deploy-mode cluster spark_code.py*
>
>                      Please can you help me with a resolution at your
> convenience.
>
> Thanks in advance !
>
> Cheers,
>
> Debu
>
>
>