You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Prem Sahoo <pr...@gmail.com> on 2025/04/11 11:57:21 UTC
SPARK ON KUBERNETS IS SLOW
Hello Team,
I have a peculiar case of Spark slowness.
I am using Minio as Object storage from where Spark reads & writes data. I
am using YARN as Master and executing a Spark job which takes ~5mins the
same job when run with Kubernetes as Master it takes ~8 mins .
I checked the Spark DAG in both and observed the same no of jobs/stages and
tasks. I am using the same machines which are being used in YARN and
Kubernetes .
one observation: when I have disabled Spark Dynamic allocation false and
assigned static allocation I can see the execution time in Kubernetes based
Spark job ~5.5 mins.
May I ask the team what could be the reason that Spark job runs slow on
kubernetes and what can be done to make it faster ?
Note :- I am using Spark 3.2 in both.
Re: SPARK ON KUBERNETS IS SLOW
Posted by karan alang <ka...@gmail.com>.
👍
karan alang reacted via Gmail
<https://www.google.com/gmail/about/?utm_source=gmail-in-product&utm_medium=et&utm_campaign=emojireactionemail#app>
On Fri, Apr 11, 2025 at 11:57 AM Prem Sahoo <pr...@gmail.com> wrote:
>
> Thanks Karan for input but found the issue of slowness in kubernetes
> while doing broadcast it takes 2 to 3 times than YARN .
> need to check why such a big difference. In our case we are doing around
> 40 tables manual broadcast as the size of tables is more than 10 mb. What
> can be done in kubernetes so that this broadcast time will be at par with
> YARN.
>
> Note : here we are using client mode in both YARN and kubernetes.
> Sent from my iPhone
>
> On Apr 11, 2025, at 2:07 PM, karan alang <ka...@gmail.com> wrote:
>
>
> Pls check if there are resource constraints on the pods/nodes especially
> if they are shared.
> MinIO connectivity performance needs to be checked.
>
> With YARN and External Spark Shuffle, the sparkshuffle is a lot more
> optimized, so we can experience slowness with spark on k8s, especially if
> there is a pod restart.
>
> Have you checked Apache Uniffle / Celeborn for enabling spark shuffle ?
>
> fyi .. i'm using kubeflow spark operator, and in the process of doing
> performance comparison/optimization as well.
>
> regds,
> Karan Alang
>
>
> On Fri, Apr 11, 2025 at 9:07 AM Prem Sahoo <pr...@gmail.com> wrote:
>
>> Hello Karan,
>> I am using Spark open source in kubernetes and Spark mapr bundle in YARN.
>>
>> For launching job in both approach it takes same 10 secs .
>>
>> For shuffle I am using local in both yarn and kubernetes.
>> Sent from my iPhone
>>
>> On Apr 11, 2025, at 11:24 AM, karan alang <ka...@gmail.com> wrote:
>>
>>
>> Hi Prem,
>>
>> Which distribution of Spark are you using ?
>> how long does it take to launch the job ?
>> wrt Spark Shuffle, what is the approach you are using - storing shuffle
>> data in MinIO or using host path ?
>>
>> regds,
>> Karan
>>
>> On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo <pr...@gmail.com> wrote:
>>
>>> Hello Team,
>>> I have a peculiar case of Spark slowness.
>>> I am using Minio as Object storage from where Spark reads & writes data.
>>> I am using YARN as Master and executing a Spark job which takes ~5mins the
>>> same job when run with Kubernetes as Master it takes ~8 mins .
>>>
>>> I checked the Spark DAG in both and observed the same no of jobs/stages
>>> and tasks. I am using the same machines which are being used in YARN and
>>> Kubernetes .
>>>
>>> one observation: when I have disabled Spark Dynamic allocation false and
>>> assigned static allocation I can see the execution time in Kubernetes based
>>> Spark job ~5.5 mins.
>>>
>>> May I ask the team what could be the reason that Spark job runs slow on
>>> kubernetes and what can be done to make it faster ?
>>> Note :- I am using Spark 3.2 in both.
>>>
>>>
Re: SPARK ON KUBERNETS IS SLOW
Posted by Prem Sahoo <pr...@gmail.com>.
Thanks Karan for input but found the issue of slowness in kubernetes while
doing broadcast it takes 2 to 3 times than YARN .
need to check why such a big difference. In our case we are doing around 40
tables manual broadcast as the size of tables is more than 10 mb. What can be
done in kubernetes so that this broadcast time will be at par with YARN.
Note : here we are using client mode in both YARN and kubernetes.
Sent from my iPhone
> On Apr 11, 2025, at 2:07 PM, karan alang <ka...@gmail.com> wrote:
>
>
>
>
> Pls check if there are resource constraints on the pods/nodes especially if
> they are shared.
>
> MinIO connectivity performance needs to be checked.
>
>
>
>
> With YARN and External Spark Shuffle, the sparkshuffle is a lot more
> optimized, so we can experience slowness with spark on k8s, especially if
> there is a pod restart.
>
>
>
> Have you checked Apache Uniffle / Celeborn for enabling spark shuffle ?
>
>
>
>
> fyi .. i'm using kubeflow spark operator, and in the process of doing
> performance comparison/optimization as well.
>
>
>
> regds,
>
> Karan Alang
>
>
>
>
>
>
>
> On Fri, Apr 11, 2025 at 9:07 AM Prem Sahoo
> <[prem.remit@gmail.com](mailto:prem.remit@gmail.com)> wrote:
>
>
>> Hello Karan,
>>
>> I am using Spark open source in kubernetes and Spark mapr bundle in YARN.
>>
>>
>
>>
>> For launching job in both approach it takes same 10 secs .
>>
>>
>
>>
>> For shuffle I am using local in both yarn and kubernetes.
>
>>
>> Sent from my iPhone
>>
>>
>
>>
>>> On Apr 11, 2025, at 11:24 AM, karan alang
<[karan.alang@gmail.com](mailto:karan.alang@gmail.com)> wrote:
>
>
>>
>>>
>>>
>>> Hi Prem,
>>>
>>>
>
>>>
>>> Which distribution of Spark are you using ?
>>>
>>> how long does it take to launch the job ?
>>>
>>> wrt Spark Shuffle, what is the approach you are using - storing shuffle
data in MinIO or using host path ?
>>>
>>>
>
>>>
>>> regds,
>>>
>>> Karan
>>>
>>>
>
>>>
>>> On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo
<[prem.remit@gmail.com](mailto:prem.remit@gmail.com)> wrote:
>
>>>
>>>> Hello Team,
>
>>>>
>>>> I have a peculiar case of Spark slowness.
>>>>
>>>> I am using Minio as Object storage from where Spark reads & writes data.
I am using YARN as Master and executing a Spark job which takes ~5mins the
same job when run with Kubernetes as Master it takes ~8 mins .
>>>>
>>>>
>
>>>>
>>>> I checked the Spark DAG in both and observed the same no of jobs/stages
and tasks. I am using the same machines which are being used in YARN and
Kubernetes .
>>>>
>>>>
>
>>>>
>>>> one observation: when I have disabled Spark Dynamic allocation false and
assigned static allocation I can see the execution time in Kubernetes based
Spark job ~5.5 mins.
>>>>
>>>>
>
>>>>
>>>> May I ask the team what could be the reason that Spark job runs slow on
kubernetes and what can be done to make it faster ?
>>>>
>>>> Note :- I am using Spark 3.2 in both.
>>>>
>>>>
>
Re: SPARK ON KUBERNETS IS SLOW
Posted by karan alang <ka...@gmail.com>.
Pls check if there are resource constraints on the pods/nodes especially if
they are shared.
MinIO connectivity performance needs to be checked.
With YARN and External Spark Shuffle, the sparkshuffle is a lot more
optimized, so we can experience slowness with spark on k8s, especially if
there is a pod restart.
Have you checked Apache Uniffle / Celeborn for enabling spark shuffle ?
fyi .. i'm using kubeflow spark operator, and in the process of doing
performance comparison/optimization as well.
regds,
Karan Alang
On Fri, Apr 11, 2025 at 9:07 AM Prem Sahoo <pr...@gmail.com> wrote:
> Hello Karan,
> I am using Spark open source in kubernetes and Spark mapr bundle in YARN.
>
> For launching job in both approach it takes same 10 secs .
>
> For shuffle I am using local in both yarn and kubernetes.
> Sent from my iPhone
>
> On Apr 11, 2025, at 11:24 AM, karan alang <ka...@gmail.com> wrote:
>
>
> Hi Prem,
>
> Which distribution of Spark are you using ?
> how long does it take to launch the job ?
> wrt Spark Shuffle, what is the approach you are using - storing shuffle
> data in MinIO or using host path ?
>
> regds,
> Karan
>
> On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo <pr...@gmail.com> wrote:
>
>> Hello Team,
>> I have a peculiar case of Spark slowness.
>> I am using Minio as Object storage from where Spark reads & writes data.
>> I am using YARN as Master and executing a Spark job which takes ~5mins the
>> same job when run with Kubernetes as Master it takes ~8 mins .
>>
>> I checked the Spark DAG in both and observed the same no of jobs/stages
>> and tasks. I am using the same machines which are being used in YARN and
>> Kubernetes .
>>
>> one observation: when I have disabled Spark Dynamic allocation false and
>> assigned static allocation I can see the execution time in Kubernetes based
>> Spark job ~5.5 mins.
>>
>> May I ask the team what could be the reason that Spark job runs slow on
>> kubernetes and what can be done to make it faster ?
>> Note :- I am using Spark 3.2 in both.
>>
>>
Re: SPARK ON KUBERNETS IS SLOW
Posted by Prem Sahoo <pr...@gmail.com>.
Hello Karan,
I am using Spark open source in kubernetes and Spark mapr bundle in YARN.
For launching job in both approach it takes same 10 secs .
For shuffle I am using local in both yarn and kubernetes.
Sent from my iPhone
> On Apr 11, 2025, at 11:24 AM, karan alang <ka...@gmail.com> wrote:
>
>
>
>
> Hi Prem,
>
>
>
>
> Which distribution of Spark are you using ?
>
> how long does it take to launch the job ?
>
> wrt Spark Shuffle, what is the approach you are using - storing shuffle data
> in MinIO or using host path ?
>
>
>
>
> regds,
>
> Karan
>
>
>
>
> On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo
> <[prem.remit@gmail.com](mailto:prem.remit@gmail.com)> wrote:
>
>
>> Hello Team,
>
>>
>> I have a peculiar case of Spark slowness.
>>
>> I am using Minio as Object storage from where Spark reads & writes data. I
am using YARN as Master and executing a Spark job which takes ~5mins the same
job when run with Kubernetes as Master it takes ~8 mins .
>>
>>
>
>>
>> I checked the Spark DAG in both and observed the same no of jobs/stages and
tasks. I am using the same machines which are being used in YARN and
Kubernetes .
>>
>>
>
>>
>> one observation: when I have disabled Spark Dynamic allocation false and
assigned static allocation I can see the execution time in Kubernetes based
Spark job ~5.5 mins.
>>
>>
>
>>
>> May I ask the team what could be the reason that Spark job runs slow on
kubernetes and what can be done to make it faster ?
>>
>> Note :- I am using Spark 3.2 in both.
>>
>>
>
Re: SPARK ON KUBERNETS IS SLOW
Posted by karan alang <ka...@gmail.com>.
Hi Prem,
Which distribution of Spark are you using ?
how long does it take to launch the job ?
wrt Spark Shuffle, what is the approach you are using - storing shuffle
data in MinIO or using host path ?
regds,
Karan
On Fri, Apr 11, 2025 at 4:58 AM Prem Sahoo <pr...@gmail.com> wrote:
> Hello Team,
> I have a peculiar case of Spark slowness.
> I am using Minio as Object storage from where Spark reads & writes data. I
> am using YARN as Master and executing a Spark job which takes ~5mins the
> same job when run with Kubernetes as Master it takes ~8 mins .
>
> I checked the Spark DAG in both and observed the same no of jobs/stages
> and tasks. I am using the same machines which are being used in YARN and
> Kubernetes .
>
> one observation: when I have disabled Spark Dynamic allocation false and
> assigned static allocation I can see the execution time in Kubernetes based
> Spark job ~5.5 mins.
>
> May I ask the team what could be the reason that Spark job runs slow on
> kubernetes and what can be done to make it faster ?
> Note :- I am using Spark 3.2 in both.
>
>