HPA - Kubernetes for Spark

[Sachit Murarka <co...@gmail.com>]

Hi All,

I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).

I understand it can be achieved by setting the request and limit for
resources in yaml:
kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1 --max=10
// example command.

But does Kubernetes actually work with Spark for HPA? Since 1 pod is used
to launch 1 unique executor. Here in spark ideally pod can be scaled by
dynamical allocation of executors(which in turn is a pod) instead of HPA.
But Dynamic allocation is not supported as shuffle service is not there
till Spark 3 release.

Could any one suggest how can I proceed achieving pod scaling in Spark?

Please note : I am using Kubernetes with Spark operator.


Kind Regards,
Sachit Murarka

Re: HPA - Kubernetes for Spark

[Sachit Murarka <co...@gmail.com>]

Hi ,

Yes I know  by setting shuffle tracking property enabled we can use DRA.
But , it is marked as experimental. Is it advised to use ?

Also , regarding HPA. We do not have HPA differently as such for Spark.
Right?

Kind Regards,
Sachit Murarka


On Mon, Jan 11, 2021 at 2:17 AM Sandish Kumar HN <sa...@gmail.com>
wrote:

> Sachit,
>
> K8S based spark dynamic allocation is only available on Spark 3.0.X+ and
> that too without External Shuffling Service.
>
>
> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
> http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work
>
> On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <co...@gmail.com>
> wrote:
>
>> Hi All,
>>
>> I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).
>>
>> I understand it can be achieved by setting the request and limit for
>> resources in yaml:
>> kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1
>> --max=10  // example command.
>>
>> But does Kubernetes actually work with Spark for HPA? Since 1 pod is used
>> to launch 1 unique executor. Here in spark ideally pod can be scaled by
>> dynamical allocation of executors(which in turn is a pod) instead of HPA.
>> But Dynamic allocation is not supported as shuffle service is not there
>> till Spark 3 release.
>>
>> Could any one suggest how can I proceed achieving pod scaling in Spark?
>>
>> Please note : I am using Kubernetes with Spark operator.
>>
>>
>> Kind Regards,
>> Sachit Murarka
>>
>
>
> --
>
> Thanks,
> Regards,
> SandishKumar HN
>

Re: HPA - Kubernetes for Spark

[Sandish Kumar HN <sa...@gmail.com>]

Sachit,

K8S based spark dynamic allocation is only available on Spark 3.0.X+ and
that too without External Shuffling Service.

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/user-guide.md#dynamic-allocation
http://spark.apache.org/docs/latest/running-on-kubernetes.html#future-work

On Sun, 10 Jan 2021 at 13:23, Sachit Murarka <co...@gmail.com>
wrote:

> Hi All,
>
> I have read about HPA  Horizontal Pod Autoscaling(for pod scaling).
>
> I understand it can be achieved by setting the request and limit for
> resources in yaml:
> kubectl autoscale deploy/application-cpu --cpu-percent=95 --min=1
> --max=10  // example command.
>
> But does Kubernetes actually work with Spark for HPA? Since 1 pod is used
> to launch 1 unique executor. Here in spark ideally pod can be scaled by
> dynamical allocation of executors(which in turn is a pod) instead of HPA.
> But Dynamic allocation is not supported as shuffle service is not there
> till Spark 3 release.
>
> Could any one suggest how can I proceed achieving pod scaling in Spark?
>
> Please note : I am using Kubernetes with Spark operator.
>
>
> Kind Regards,
> Sachit Murarka
>


-- 

Thanks,
Regards,
SandishKumar HN