Spark on YARN in client-mode: do we need 1 vCore for the AM?

[peay <pe...@protonmail.com>]

Hello,

I run a Spark cluster on YARN, and we have a bunch of client-mode applications we use for interactive work. Whenever we start one of this, an application master container is started.

My understanding is that this is mostly an empty shell, used to request further containers or get status from YARN. Is that correct?

spark.yarn.am.cores is 1, and that AM gets one full vCore on the cluster. Because I am using DominantResourceCalculator to take vCores into account for scheduling, this results in a lot of unused CPU capacity overall because all those AMs each block one full vCore. With enough jobs, this adds up quickly.

I am trying to understand if we can work around that -- ideally, by allocating fractional vCores (e.g., give 100 millicores to the AM), or by allocating no vCores at all for the AM (I am fine with a bit of oversubscription because of that).

Any idea on how to avoid blocking so many YARN vCores just for the Spark AMs?

Thanks!

Re: Spark on YARN in client-mode: do we need 1 vCore for the AM?

[Jeff Zhang <zj...@gmail.com>]

I don't think it is possible to have less than 1 core for AM, this is due
to yarn not spark.

The number of AM comparing to the number of executors should be small and
acceptable. If you do want to save more resources, I would suggest you to
use yarn cluster mode where driver and AM run in the same process.

You can either use livy or zeppelin which both support interactive work in
yarn cluster mode.

http://livy.incubator.apache.org/
https://zeppelin.apache.org/
https://medium.com/@zjffdu/zeppelin-0-8-0-new-features-ea53e8810235


Another approach to save resources is to share SparkContext across your
applications since your scenario is interactive work ( I guess it is some
kind of notebook).  Zeppelin support sharing SparkContext across users and
notes.



peay <pe...@protonmail.com>于2018年5月18日周五 下午6:20写道：

> Hello,
>
> I run a Spark cluster on YARN, and we have a bunch of client-mode
> applications we use for interactive work. Whenever we start one of this, an
> application master container is started.
>
> My understanding is that this is mostly an empty shell, used to request
> further containers or get status from YARN. Is that correct?
>
> spark.yarn.am.cores is 1, and that AM gets one full vCore on the cluster.
> Because I am using DominantResourceCalculator to take vCores into account
> for scheduling, this results in a lot of unused CPU capacity overall
> because all those AMs each block one full vCore. With enough jobs, this
> adds up quickly.
>
> I am trying to understand if we can work around that -- ideally, by
> allocating fractional vCores (e.g., give 100 millicores to the AM), or by
> allocating no vCores at all for the AM (I am fine with a bit of
> oversubscription because of that).
>
> Any idea on how to avoid blocking so many YARN vCores just for the Spark
> AMs?
>
> Thanks!
>
>