Apache Spark Serverless pool and preemptive execution support

[Mania Abdi <ab...@husky.neu.edu>]

Hi everyone,

I came across Databricks serverless analytics on Spark
<https://databricks.com/blog/2017/06/07/databricks-serverless-next-generation-resource-management-for-apache-spark.html>and
Serverless pools. I was wondering if the Apache Spark also supports these
pools and if the Apache Spark resource manager supports
preemptive execution and auto-scaling?

Regards
Mania

Re: Apache Spark Serverless pool and preemptive execution support

[Sean Owen <sr...@gmail.com>]

The built-in resource managers do support dynamic allocation for
auto-scaling.
https://spark.apache.org/docs/latest/configuration.html#dynamic-allocation

Provisioning the resources for the resource manager is a different question
that Spark itself doesn't address; it won't go run EC2/Azure instances for
you, but gives you the resource manager to run on those. The 'serverless'
part you refer to here is really answering the question of provisioning VMs
for a particular hosted Spark service. That isn't part of Spark per se.


On Sat, Mar 28, 2020 at 10:38 AM Mania Abdi <ab...@husky.neu.edu> wrote:

> Hi everyone,
>
> I came across Databricks serverless analytics on Spark
> <https://databricks.com/blog/2017/06/07/databricks-serverless-next-generation-resource-management-for-apache-spark.html>and
> Serverless pools. I was wondering if the Apache Spark also supports these
> pools and if the Apache Spark resource manager supports
> preemptive execution and auto-scaling?
>
> Regards
> Mania
>