Running Hive and Spark together with Dynamic Resource Allocation

[rachmaninovquartet <ra...@gmail.com>]

Hi,

My team has a cluster running HDP, with Hive and Spark. We setup spark to
use dynamic resource allocation, for benefits such as not having to hard
code the number of executors and to free resources after using. Everything
is running on YARN.

The problem is that for Spark 1.5.2 with dynamic resource allocation to
function properly we needed to set yarn.nodemanager.aux-services in
yarn-site.xml to spark_shuffle, but this breaks hive (1.2.1), since it is
looking for auxService:mapreduce_shuffle. Does any one know of a way to
configure in order to have both services running smoothly?

Thanks,

Ian



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Hive-and-Spark-together-with-Dynamic-Resource-Allocation-tp27968.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Running Hive and Spark together with Dynamic Resource Allocation

[rachmaninovquartet <ra...@gmail.com>]

It seems like the best solution is to set: yarn.nodemanager.aux-services to
mapred_shuffle,spark_shuffle



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Running-Hive-and-Spark-together-with-Dynamic-Resource-Allocation-tp27968p27978.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org

Re: Running Hive and Spark together with Dynamic Resource Allocation

[Stéphane Verlet <ka...@gmail.com>]

This works for us

 <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle,spark_shuffle</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

  <property>
    <name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
    <value>org.apache.spark.network.yarn.YarnShuffleService</value>
  </property>





On Thu, Oct 27, 2016 at 8:13 AM, rachmaninovquartet <
rachmaninovquartet@gmail.com> wrote:

> oblem is that for Spark 1.5.2 with dynamic resource allocation to
> function properly we needed to set y
>