You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Rajesh Mahindra <rj...@gmail.com> on 2024/06/26 22:35:39 UTC

Spark decommission

Hi folks,

I am planning to leverage the "Spark Decommission" feature in production
since our company uses SPOT instances on Kubernetes. I wanted to get a
sense of how stable the feature is for production usage and if any one has
thoughts around trying it out in production, especially in kubernetes
environment.

Thanks,
Rajesh

Re: Spark decommission

Posted by Steve Loughran <st...@cloudera.com.INVALID>.
On Fri, 5 Jul 2024 at 01:44, Arun Ravi <ar...@gmail.com> wrote:

> Hi Rajesh,
>
> We use it production at scale. We run spark on kubernetes on aws cloud and
> here are the key things that we do
> 1) we run driver on on-demand node
> 2) we have configured decommission along with fallback option on to S3,
> try the latest single zone S3 for this.
>

you mean s3 express? through the s3a connector ?

if so I'd love to know your performance statistics which you can get
printed when a process exits

fs.iostatistics.logging.level = info

if you are actually reading data from it -Upgrade to parquet 14.1 and turn
on vector IO; it really takes advantage of the low latency and improved
bandwidth.

> 3) We use pvc aware scheduling, ie spark ensures executors try to reuse
> available storage volumes created by the driver before requesting for a new
> one.
> 4) we have enabled kubernetes shuffle io wrapper plugin, this allows new
> executors to re-register shuffle blocks that it identifies in the reused
> pvc. This feature ensures shuffles from lost executors are served by new
> executor that refuses the disk.
> 5) we also configure to retain decommissioned executor details so that
> spark can ignore intermittent shuffle fetch failures.
>
> Some of these are best effort, you could also tune number of threads
> needed for decommissioning etc based on your workload and run environment.
>
> On Thu, 27 Jun 2024, 09:03 Rajesh Mahindra, <rj...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I am planning to leverage the "Spark Decommission" feature in production
>> since our company uses SPOT instances on Kubernetes. I wanted to get a
>> sense of how stable the feature is for production usage and if any one has
>> thoughts around trying it out in production, especially in kubernetes
>> environment.
>>
>> Thanks,
>> Rajesh
>>
>>

Re: Spark decommission

Posted by Arun Ravi <ar...@gmail.com>.
Hi Rajesh,

We use it production at scale. We run spark on kubernetes on aws cloud and
here are the key things that we do
1) we run driver on on-demand node
2) we have configured decommission along with fallback option on to S3, try
the latest single zone S3 for this.
3) We use pvc aware scheduling, ie spark ensures executors try to reuse
available storage volumes created by the driver before requesting for a new
one.
4) we have enabled kubernetes shuffle io wrapper plugin, this allows new
executors to re-register shuffle blocks that it identifies in the reused
pvc. This feature ensures shuffles from lost executors are served by new
executor that refuses the disk.
5) we also configure to retain decommissioned executor details so that
spark can ignore intermittent shuffle fetch failures.

Some of these are best effort, you could also tune number of threads needed
for decommissioning etc based on your workload and run environment.

On Thu, 27 Jun 2024, 09:03 Rajesh Mahindra, <rj...@gmail.com> wrote:

> Hi folks,
>
> I am planning to leverage the "Spark Decommission" feature in production
> since our company uses SPOT instances on Kubernetes. I wanted to get a
> sense of how stable the feature is for production usage and if any one has
> thoughts around trying it out in production, especially in kubernetes
> environment.
>
> Thanks,
> Rajesh
>
>