Re: Executors assigned to STS and number of workers in Stand Alone Mode

[Michael Gummelt <mg...@mesosphere.io>]

> but Spark on Mesos is certainly lagging behind Spark on YARN regarding
the features Spark uses off the scheduler backends -- security, data
locality, queues, etc.

If by security you mean Kerberos, we'll be upstreaming that to Apache Spark
soon.  It's been in DC/OS Spark for a while:
https://github.com/mesosphere/spark/commit/73ba2ab8d97510d5475ef9a48c673ce34f7173fa

Locality is implemented in a scheduler independent way:
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala#L327,
but it is possible that the offer model could result in different
placement.  I haven't seen any analysis to that effect.

YARN queues are very similar to Mesos quota and roles, which Spark
supports.  We'll also be adding support for revocable resource support
sometime soon, which solves the HoL blocking problem, where one Spark app
eats up your cluster while others wait.  I don't think YARN has a solution
for this, but I could be wrong.

So, yea, there are some differences, but I think the biggest feature gap
right now is really just Kerberos, which will be added soon.

There are also other Mesos-specific features we'll be adding soon, such as
GPU, CNI, and virtual network but the biggest advantage for running on
Mesos is that you can run multi-tenant alongside other Mesos frameworks.








On Mon, Jul 25, 2016 at 2:04 PM, Jacek Laskowski <ja...@japila.pl> wrote:

> On Mon, Jul 25, 2016 at 10:57 PM, Mich Talebzadeh
> <mi...@gmail.com> wrote:
>
> > Yarn promises the best resource management I believe. Having said that I
> have not used Mesos myself.
>
> I'm glad you've mentioned it.
>
> I think Cloudera (and Hortonworks?) guys are doing a great job with
> bringing all the features of YARN to Spark and I think Spark on YARN
> shines features-wise.
>
> I'm not in a position to compare YARN vs Mesos for their resource
> management, but Spark on Mesos is certainly lagging behind Spark on
> YARN regarding the features Spark uses off the scheduler backends --
> security, data locality, queues, etc. (or I might be simply biased
> after having spent months with Spark on YARN mostly?).
>
> Jacek
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>


-- 
Michael Gummelt
Software Engineer
Mesosphere