You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Jahar Tyagi <ja...@gmail.com> on 2021/02/16 11:27:10 UTC

vm.swappiness value for Spark on Kubernetes

Hi,

We have recently migrated from Spark 2.4.4 to Spark 3.0.1 and using Spark
in virtual machine/bare metal as standalone deployment and as kubernetes
deployment as well.

There is a kernel parameter named as 'vm.swappiness' and we keep its value
as '1' in standard deployment. Now since we are moving to kubernetes and on
kubernetes worker nodes the value of this parameter is '60'.

Now my question is if it is OK to keep such a high value of
'vm.swappiness'=60 in kubernetes environment for Spark workloads.

Will such high value of this kernel parameter have performance impact on
Spark PODs?
As per below link from cloudera, they suggest not to set such a high value.

https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-setting-vmswappiness-linux-kernel-parameter.html

Any thoughts/suggestions on this are highly appreciated.

Regards
Jahar Tyagi

Re: vm.swappiness value for Spark on Kubernetes

Posted by Sean Owen <sr...@gmail.com>.
You probably don't want swapping in any environment. Some tasks will grind
to a halt under mem pressure rather than just fail quickly. You would want
to simply provision more memory.

On Tue, Feb 16, 2021, 7:57 AM Jahar Tyagi <ja...@gmail.com> wrote:

> Hi,
>
> We have recently migrated from Spark 2.4.4 to Spark 3.0.1 and using Spark
> in virtual machine/bare metal as standalone deployment and as kubernetes
> deployment as well.
>
> There is a kernel parameter named as 'vm.swappiness' and we keep its value
> as '1' in standard deployment. Now since we are moving to kubernetes and on
> kubernetes worker nodes the value of this parameter is '60'.
>
> Now my question is if it is OK to keep such a high value of
> 'vm.swappiness'=60 in kubernetes environment for Spark workloads.
>
> Will such high value of this kernel parameter have performance impact on
> Spark PODs?
> As per below link from cloudera, they suggest not to set such a high
> value.
>
>
> https://docs.cloudera.com/cloudera-manager/7.2.6/managing-clusters/topics/cm-setting-vmswappiness-linux-kernel-parameter.html
>
> Any thoughts/suggestions on this are highly appreciated.
>
> Regards
> Jahar Tyagi
>
>