You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by jamborta <ja...@gmail.com> on 2014/09/28 03:38:09 UTC

yarn does not accept job in cluster mode

hi all,

I have a job that works ok in yarn-client mode,but when I try in
yarn-cluster mode it returns the following:

WARN YarnClusterScheduler: Initial job has not accepted any resources; check
your cluster UI to ensure that workers are registered and have sufficient
memory

the cluster has plenty of memory and resources. I am running this from
python using this context:

    conf = (SparkConf()
            .setMaster("yarn-cluster")
            .setAppName("spark_tornado_server")
            .set("spark.executor.memory", "1024m")
            .set("spark.cores.max", 16)
            .set("spark.driver.memory", "1024m")
            .set("spark.executor.instances", 2)
            .set("spark.executor.cores", 8)
            .set("spark.eventLog.enabled", False)

HADOOP_HOME and HADOOP_CONF_DIR are also set in spark-env.

thanks,

not sure if I am missing some config




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/yarn-does-not-accept-job-in-cluster-mode-tp15281.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: yarn does not accept job in cluster mode

Posted by Tamas Jambor <ja...@gmail.com>.

thanks for the reply.

As I mentioned above, all works in yarn-client mode, the problem
starts when I try to run it in yarn-cluster mode.

(seems that spark-shell does not work in yarn-cluster mode, so cannot
debug that way).


On Mon, Sep 29, 2014 at 7:30 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> Can you try running the spark-shell in yarn-cluster mode?
>
> ./bin/spark-shell --master yarn-client
>
> Read more over here http://spark.apache.org/docs/1.0.0/running-on-yarn.html
>
> Thanks
> Best Regards
>
> On Sun, Sep 28, 2014 at 7:08 AM, jamborta <ja...@gmail.com> wrote:
>>
>> hi all,
>>
>> I have a job that works ok in yarn-client mode,but when I try in
>> yarn-cluster mode it returns the following:
>>
>> WARN YarnClusterScheduler: Initial job has not accepted any resources;
>> check
>> your cluster UI to ensure that workers are registered and have sufficient
>> memory
>>
>> the cluster has plenty of memory and resources. I am running this from
>> python using this context:
>>
>>     conf = (SparkConf()
>>             .setMaster("yarn-cluster")
>>             .setAppName("spark_tornado_server")
>>             .set("spark.executor.memory", "1024m")
>>             .set("spark.cores.max", 16)
>>             .set("spark.driver.memory", "1024m")
>>             .set("spark.executor.instances", 2)
>>             .set("spark.executor.cores", 8)
>>             .set("spark.eventLog.enabled", False)
>>
>> HADOOP_HOME and HADOOP_CONF_DIR are also set in spark-env.
>>
>> thanks,
>>
>> not sure if I am missing some config
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/yarn-does-not-accept-job-in-cluster-mode-tp15281.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: yarn does not accept job in cluster mode

Posted by Akhil Das <ak...@sigmoidanalytics.com>.

Can you try running the spark-shell in yarn-cluster mode?

./bin/spark-shell --master yarn-client

Read more over here http://spark.apache.org/docs/1.0.0/running-on-yarn.html

Thanks
Best Regards

On Sun, Sep 28, 2014 at 7:08 AM, jamborta <ja...@gmail.com> wrote:

> hi all,
>
> I have a job that works ok in yarn-client mode,but when I try in
> yarn-cluster mode it returns the following:
>
> WARN YarnClusterScheduler: Initial job has not accepted any resources;
> check
> your cluster UI to ensure that workers are registered and have sufficient
> memory
>
> the cluster has plenty of memory and resources. I am running this from
> python using this context:
>
>     conf = (SparkConf()
>             .setMaster("yarn-cluster")
>             .setAppName("spark_tornado_server")
>             .set("spark.executor.memory", "1024m")
>             .set("spark.cores.max", 16)
>             .set("spark.driver.memory", "1024m")
>             .set("spark.executor.instances", 2)
>             .set("spark.executor.cores", 8)
>             .set("spark.eventLog.enabled", False)
>
> HADOOP_HOME and HADOOP_CONF_DIR are also set in spark-env.
>
> thanks,
>
> not sure if I am missing some config
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/yarn-does-not-accept-job-in-cluster-mode-tp15281.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>