You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2022/01/25 19:44:06 UTC

Spark and scalability in k8s etc

Hi,

In a nutshell what is the role of Spark and the cluster manager on
scalability.

For example, we have recently seen the addition of auto-scaling to Google
Dataproc
<https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling>
and the availability of autoscaling with the Kubernetes engine.

But how does this architecturally work? The current k8s on Spark model is
constrained to one executor per pod and one pod per node. I recently
discussed this and my findings were that  you can fit one Spark executor pod
 per Kubernetes node. Let us assume this is the case. If this is true then
from classic spark we know that if one executor fails, it will move the
processing over to the other executor. However, if the data is lost, it
re-executes the processing that generated the data. We also know that there
will be finite time for another node to be brought up online. In this case
how will spark handle this? Will it wait for the current tasks (including
anything that belonged to the tasks on the failed node which were moved to
the existing executors) to complete before utilising the newly created
executor on the newly created node?



[image: image.png]


As labelled in the diagram above, this is the way I believe spark on k8s
work.


   1.  processes 1-3 will be dealing with the request for Kube apiserver to
   schedule the creation of driver pod and at the completion, process 4
   originating from the driver will request the kube apiserver to schedule
   creation of executors.
   2. Depending on the number of executors requested through
   spark.executor.instances property, the driver will ask for these in chunks
   i.e. it will ask for N executors, wait briefly and then ask for another N
   executors. Where N is a controllable batch size (default 5) managed via the
   *spark.kubernetes.allocation.batch.size* property and the delay between
   batches of executors (default 1s) controlled by the
   spark.kubernetes.allocation.batch.delay property. This batch based
   allocation is done to avoid overwhelming the Kube apiserver and effectively
   overloading the cluster by asking for all the pods at once.
   3. Processes 5 and 6 will be dealing with creating executors through the
   scheduler.

Will the Kube apiserver request the scheduler to add a new Executor Pod on
the new node, what will it contribute in the middle of a running job?

Thanks


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.

Re: Spark and scalability in k8s etc

Posted by Mich Talebzadeh <mi...@gmail.com>.
Thanks for the info provided.

In this model of yours where does your driver sit? The reason I am asking
this is that k8s creates the driver first and the driver in turn instructs
Kube apierver to schedule executors. Your configuration above does not
mention anything about the driver memory and cores?

HTH



   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 26 Jan 2022 at 13:06, Bjørn Jørgensen <bj...@gmail.com>
wrote:

> “The current k8s on Spark model is constrained to one executor per pod and
> one pod per node. I recently discussed this and my findings were that you
> can fit one Spark executor pod per Kubernetes node. Let us assume this is
> the case.”
>
>
>
> With my experience you can have as many pods per node as you want to.
>
> If you request 10 pods and the cluster has the resources available, K8S
> will give you 10 pods on 1 node.
>
>
>
> So, if you have one node with 5 available cores and 5 GB RAM available.
>
> You can request 5 executors with each pod memory set to 1g and cores set
> to 1.
>
> .set("spark.executor.instances", "5")
>
> .set("spark.executor.memory", "1g")
>
> .set("spark.executor.cores", "1")
>
> K8S will then start 5 executors on that node.
>
> The important thing here is that a node must have the requested resources
> available.
>
> tir. 25. jan. 2022 kl. 21:00 skrev Mich Talebzadeh <
> mich.talebzadeh@gmail.com>:
>
>> Hi,
>>
>> In a nutshell what is the role of Spark and the cluster manager on
>> scalability.
>>
>> For example, we have recently seen the addition of auto-scaling to
>> Google Dataproc
>> <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling>
>> and the availability of autoscaling with the Kubernetes engine.
>>
>> But how does this architecturally work? The current k8s on Spark model is
>> constrained to one executor per pod and one pod per node. I recently
>> discussed this and my findings were that  you can fit one Spark executor
>> pod per Kubernetes node. Let us assume this is the case. If this is true
>> then from classic spark we know that if one executor fails, it will move
>> the processing over to the other executor. However, if the data is lost, it
>> re-executes the processing that generated the data. We also know that there
>> will be finite time for another node to be brought up online. In this case
>> how will spark handle this? Will it wait for the current tasks (including
>> anything that belonged to the tasks on the failed node which were moved to
>> the existing executors) to complete before utilising the newly created
>> executor on the newly created node?
>>
>>
>>
>> [image: image.png]
>>
>>
>> As labelled in the diagram above, this is the way I believe spark on k8s
>> work.
>>
>>
>>    1.  processes 1-3 will be dealing with the request for Kube apiserver
>>    to schedule the creation of driver pod and at the completion, process 4
>>    originating from the driver will request the kube apiserver to schedule
>>    creation of executors.
>>    2. Depending on the number of executors requested through
>>    spark.executor.instances property, the driver will ask for these in chunks
>>    i.e. it will ask for N executors, wait briefly and then ask for another N
>>    executors. Where N is a controllable batch size (default 5) managed via the
>>    *spark.kubernetes.allocation.batch.size* property and the delay
>>    between batches of executors (default 1s) controlled by the
>>    spark.kubernetes.allocation.batch.delay property. This batch based
>>    allocation is done to avoid overwhelming the Kube apiserver and effectively
>>    overloading the cluster by asking for all the pods at once.
>>    3. Processes 5 and 6 will be dealing with creating executors through
>>    the scheduler.
>>
>> Will the Kube apiserver request the scheduler to add a new Executor Pod
>> on the new node, what will it contribute in the middle of a running job?
>>
>> Thanks
>>
>>
>>    view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>

Re: Spark and scalability in k8s etc

Posted by Bjørn Jørgensen <bj...@gmail.com>.
“The current k8s on Spark model is constrained to one executor per pod and
one pod per node. I recently discussed this and my findings were that you
can fit one Spark executor pod per Kubernetes node. Let us assume this is
the case.”



With my experience you can have as many pods per node as you want to.

If you request 10 pods and the cluster has the resources available, K8S
will give you 10 pods on 1 node.



So, if you have one node with 5 available cores and 5 GB RAM available.

You can request 5 executors with each pod memory set to 1g and cores set to
1.

.set("spark.executor.instances", "5")

.set("spark.executor.memory", "1g")

.set("spark.executor.cores", "1")

K8S will then start 5 executors on that node.

The important thing here is that a node must have the requested resources
available.

tir. 25. jan. 2022 kl. 21:00 skrev Mich Talebzadeh <
mich.talebzadeh@gmail.com>:

> Hi,
>
> In a nutshell what is the role of Spark and the cluster manager on
> scalability.
>
> For example, we have recently seen the addition of auto-scaling to Google
> Dataproc
> <https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/autoscaling>
> and the availability of autoscaling with the Kubernetes engine.
>
> But how does this architecturally work? The current k8s on Spark model is
> constrained to one executor per pod and one pod per node. I recently
> discussed this and my findings were that  you can fit one Spark executor
> pod per Kubernetes node. Let us assume this is the case. If this is true
> then from classic spark we know that if one executor fails, it will move
> the processing over to the other executor. However, if the data is lost, it
> re-executes the processing that generated the data. We also know that there
> will be finite time for another node to be brought up online. In this case
> how will spark handle this? Will it wait for the current tasks (including
> anything that belonged to the tasks on the failed node which were moved to
> the existing executors) to complete before utilising the newly created
> executor on the newly created node?
>
>
>
> [image: image.png]
>
>
> As labelled in the diagram above, this is the way I believe spark on k8s
> work.
>
>
>    1.  processes 1-3 will be dealing with the request for Kube apiserver
>    to schedule the creation of driver pod and at the completion, process 4
>    originating from the driver will request the kube apiserver to schedule
>    creation of executors.
>    2. Depending on the number of executors requested through
>    spark.executor.instances property, the driver will ask for these in chunks
>    i.e. it will ask for N executors, wait briefly and then ask for another N
>    executors. Where N is a controllable batch size (default 5) managed via the
>    *spark.kubernetes.allocation.batch.size* property and the delay
>    between batches of executors (default 1s) controlled by the
>    spark.kubernetes.allocation.batch.delay property. This batch based
>    allocation is done to avoid overwhelming the Kube apiserver and effectively
>    overloading the cluster by asking for all the pods at once.
>    3. Processes 5 and 6 will be dealing with creating executors through
>    the scheduler.
>
> Will the Kube apiserver request the scheduler to add a new Executor Pod on
> the new node, what will it contribute in the middle of a running job?
>
> Thanks
>
>
>    view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>


-- 
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge

+47 480 94 297