You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Shivaram Venkataraman <sh...@eecs.berkeley.edu> on 2015/06/04 01:32:53 UTC

[ANNOUNCE] YARN support in Spark EC2

Hi all

We recently merged support for launching YARN clusters using Spark EC2
scripts as a part of
https://issues.apache.org/jira/browse/SPARK-3674. To use this you can pass
in hadoop-major-version as "yarn" to the spark-ec2 script and this will
setup Hadoop 2.4 HDFS, YARN and Spark built for YARN on the EC2 cluster.

Developers who work on features related to YARN might find this useful for
testing / benchmarking Spark with YARN. If anyone has questions or feedback
please let me know.

Thanks
Shivaram