You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by alvarobrandon <al...@gmail.com> on 2016/02/26 12:14:30 UTC
Dynamic allocation Spark
Hello everyone:
I'm trying the dynamic allocation in Spark with YARN. I have followed the
following configuration steps:
1. Copy the spark-*-yarn-shuffle.jar to the nodemanager classpath. "cp
/opt/spark/lib/spark-*-yarn-shuffle.jar /opt/hadoop/share/hadoop/yarn"
2. Added the shuffle service of spark in yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle</value>
<description>shuffle implementation</description>
</property>
3. Enabled the class for the shuffle service in yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
<description>enable the class for dynamic allocation</description>
</property>
4. Activated the dynamic allocation in the spark defaults
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
When I launch my application it just stays in the queue accepted but it
never actually runs.
16/02/26 11:11:46 INFO yarn.Client: Application report for
application_1456482268159_0001 (state: ACCEPTED)
Am I missing something?
Thanks in advance as always
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Dynamic-allocation-Spark-tp26344.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Dynamic allocation Spark
Posted by Alvaro Brandon <al...@gmail.com>.
That was exactly it. I had the worker and master processes of Spark
standalone running together with YARN and somehow the resource manager
didn't see the nodes. It's working now.
Thanks for the tip :-)
2016-02-26 12:33 GMT+01:00 Jeff Zhang <zj...@gmail.com>:
> Check the RM UI to ensure you have available resources. I suspect it might
> be that you didn't configure yarn correctly, so NM didn't start properly
> and you have no resource.
>
> On Fri, Feb 26, 2016 at 7:14 PM, alvarobrandon <al...@gmail.com>
> wrote:
>
>> Hello everyone:
>>
>> I'm trying the dynamic allocation in Spark with YARN. I have followed the
>> following configuration steps:
>> 1. Copy the spark-*-yarn-shuffle.jar to the nodemanager classpath. "cp
>> /opt/spark/lib/spark-*-yarn-shuffle.jar /opt/hadoop/share/hadoop/yarn"
>> 2. Added the shuffle service of spark in yarn-site.xml
>> <property>
>> <name>yarn.nodemanager.aux-services</name>
>> <value>mapreduce_shuffle,spark_shuffle</value>
>> <description>shuffle implementation</description>
>> </property>
>> 3. Enabled the class for the shuffle service in yarn-site.xml
>> <property>
>> <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
>> <value>org.apache.spark.network.yarn.YarnShuffleService</value>
>> <description>enable the class for dynamic allocation</description>
>> </property>
>> 4. Activated the dynamic allocation in the spark defaults
>> spark.dynamicAllocation.enabled true
>> spark.shuffle.service.enabled true
>>
>> When I launch my application it just stays in the queue accepted but it
>> never actually runs.
>> 16/02/26 11:11:46 INFO yarn.Client: Application report for
>> application_1456482268159_0001 (state: ACCEPTED)
>>
>> Am I missing something?
>>
>> Thanks in advance as always
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Dynamic-allocation-Spark-tp26344.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
Re: Dynamic allocation Spark
Posted by Jeff Zhang <zj...@gmail.com>.
Check the RM UI to ensure you have available resources. I suspect it might
be that you didn't configure yarn correctly, so NM didn't start properly
and you have no resource.
On Fri, Feb 26, 2016 at 7:14 PM, alvarobrandon <al...@gmail.com>
wrote:
> Hello everyone:
>
> I'm trying the dynamic allocation in Spark with YARN. I have followed the
> following configuration steps:
> 1. Copy the spark-*-yarn-shuffle.jar to the nodemanager classpath. "cp
> /opt/spark/lib/spark-*-yarn-shuffle.jar /opt/hadoop/share/hadoop/yarn"
> 2. Added the shuffle service of spark in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services</name>
> <value>mapreduce_shuffle,spark_shuffle</value>
> <description>shuffle implementation</description>
> </property>
> 3. Enabled the class for the shuffle service in yarn-site.xml
> <property>
> <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
> <value>org.apache.spark.network.yarn.YarnShuffleService</value>
> <description>enable the class for dynamic allocation</description>
> </property>
> 4. Activated the dynamic allocation in the spark defaults
> spark.dynamicAllocation.enabled true
> spark.shuffle.service.enabled true
>
> When I launch my application it just stays in the queue accepted but it
> never actually runs.
> 16/02/26 11:11:46 INFO yarn.Client: Application report for
> application_1456482268159_0001 (state: ACCEPTED)
>
> Am I missing something?
>
> Thanks in advance as always
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Dynamic-allocation-Spark-tp26344.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
--
Best Regards
Jeff Zhang