You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Ismail Yenigul <is...@gmail.com> on 2023/03/10 17:38:37 UTC
spark executor pod has same memory value for request and limit
Hi,
There is a cpu parameter to set spark executor on k8s
spark.kubernetes.executor.limit.cores and
spark.kubernetes.executor.request.cores
but there is no parameter to set memory request different then limits
memory (such as spark.kubernetes.executor.request.memory)
For that reason,
spark.executor.memory is assigned to requests.memory and limits.memory
like the following
Limits:
memory: 5734MiRequests:
cpu: 4
memory: 5734Mi
Is there any special reason to not have
spark.kubernetes.executor.request.memory parameter?
and can I use spark.kubernetes.executor.podTemplateFile parameter to set
smaller memory request than the memory limit in pod template file?
Limits:
memory: 5734MiRequests:
cpu: 4
memory: 1024Mi
Thanks
Re: spark executor pod has same memory value for request and limit
Posted by Martin Andersson <ma...@kambi.com>.
There is a very good reason for this. It is recommended using k8s that you set memory request and limit to the same value, set a cpu request, but not a cpu limit. More info here https://home.robusta.dev/blog/kubernetes-memory-limit
BR, Martin
________________________________
From: Mich Talebzadeh <mi...@gmail.com>
Sent: Friday, March 10, 2023 20:25
To: Ismail Yenigul <is...@gmail.com>
Cc: dev@spark.apache.org <de...@spark.apache.org>
Subject: Re: spark executor pod has same memory value for request and limit
EXTERNAL SENDER. Do not click links or open attachments unless you recognize the sender and know the content is safe. DO NOT provide your username or password.
agreed. need to be enhanced!
HTH
[https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0>
https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0>
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Fri, 10 Mar 2023 at 19:15, Ismail Yenigul <is...@gmail.com>> wrote:
Hi Mich,
The issue is here there is no parameter to set executor pod request memory value.
Currently we have only one parameter which is spark.executor.memory and it set pod resources limit and requests.
Mich Talebzadeh <mi...@gmail.com>>, 10 Mar 2023 Cum, 22:04 tarihinde şunu yazdı:
Yes, both EKS and GKE (Google) are on 3.1.2 so I am not sure those parameters will work :(
[https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0>
https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0>
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Fri, 10 Mar 2023 at 19:01, Ismail Yenigul <is...@gmail.com>> wrote:
Hi Mich,
it is on AWS EKS
Mich Talebzadeh <mi...@gmail.com>>, 10 Mar 2023 Cum, 21:11 tarihinde şunu yazdı:
I forgot top ask which k8s cluster are you using, assuming some clod vendor
[https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0>
https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PFJALy5l0wwTnWq8bXi59mLiH1YaRQsjlpLB%2FH04VPY%3D&reserved=0>
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <is...@gmail.com>> wrote:
and If you look at the code
https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fspark%2Fblob%2Fe64262f417bf381bdc664dfd1cbcfaa5aa7221fe%2Fresource-managers%2Fkubernetes%2Fcore%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fdeploy%2Fk8s%2Ffeatures%2FBasicExecutorFeatureStep.scala%23L194&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=j1uFQjXYZYArgc7JP5SN88lcs3paDvUCa7JEcl%2FyHko%3D&reserved=0>
.editOrNewResources()
.addToRequests("memory", executorMemoryQuantity)
.addToLimits("memory", executorMemoryQuantity)
.addToRequests("cpu", executorCpuQuantity)
.addToLimits(executorResourceQuantities.asJava)
.endResources()
addToRequests and addToLimits for memory have the same value.
maybe it is by design. but can I set custom values for them if I use podtemplate?
Ismail Yenigul <is...@gmail.com>>, 10 Mar 2023 Cum, 20:52 tarihinde şunu yazdı:
Hi,
using spark version v.3.1.2
spark.executor.memory is set.
But the problem is not setting spark.executor.memory, the problem is that whatever value I set spark.executor.memory,
spark executor pod has the same value for resources.limit.memory and resources.request.memory.
I want to be able to set different values for them.
Mich Talebzadeh <mi...@gmail.com>>, 10 Mar 2023 Cum, 20:44 tarihinde şunu yazdı:
What are those currently set in spark-submit and which spark version on k8s
--conf spark.driver.memory=2000m \
--conf spark.executor.memory=2000m \
HTH
[https://ci3.googleusercontent.com/mail-sig/AIorK4zholKucR2Q9yMrKbHNn-o1TuS4mYXyi2KO6Xmx6ikHPySa9MLaLZ8t2hrA6AUcxSxDgHIwmKE] view my Linkedin profile<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fmich-talebzadeh-ph-d-5205b2%2F&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532699317%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QFeuv8QaKxfTbMhwQ78Bdwh%2Bagfb1m9CvLF%2BFNmPtLQ%3D&reserved=0>
https://en.everybodywiki.com/Mich_Talebzadeh<https://gbr01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fen.everybodywiki.com%2FMich_Talebzadeh&data=05%7C01%7Cmartin.andersson%40kambi.com%7C2b3e255305854678fef608db219d7e01%7Ce3ec1ec4b9944e9e82e080234621871f%7C0%7C0%7C638140732532855035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=7btUJdT1bquZF%2F%2FOGTWuHeHCw94KA7AwUDAHZgIqPsM%3D&reserved=0>
Disclaimer: Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction.
On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>> wrote:
Hi,
There is a cpu parameter to set spark executor on k8s
spark.kubernetes.executor.limit.cores and spark.kubernetes.executor.request.cores
but there is no parameter to set memory request different then limits memory (such as spark.kubernetes.executor.request.memory)
For that reason,
spark.executor.memory is assigned to requests.memory and limits.memory like the following
Limits:
memory: 5734Mi
Requests:
cpu: 4
memory: 5734Mi
Is there any special reason to not have spark.kubernetes.executor.request.memory parameter?
and can I use spark.kubernetes.executor.podTemplateFile parameter to set smaller memory request than the memory limit in pod template file?
Limits:
memory: 5734Mi
Requests:
cpu: 4
memory: 1024Mi
Thanks
Re: spark executor pod has same memory value for request and limit
Posted by Mich Talebzadeh <mi...@gmail.com>.
agreed. need to be enhanced!
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 10 Mar 2023 at 19:15, Ismail Yenigul <is...@gmail.com>
wrote:
> Hi Mich,
>
> The issue is here there is no parameter to set executor pod request memory
> value.
> Currently we have only one parameter which is spark.executor.memory and it
> set pod resources limit and requests.
>
> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 22:04
> tarihinde şunu yazdı:
>
>> Yes, both EKS and GKE (Google) are on 3.1.2 so I am not sure those
>> parameters will work :(
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 10 Mar 2023 at 19:01, Ismail Yenigul <is...@gmail.com>
>> wrote:
>>
>>> Hi Mich,
>>>
>>> it is on AWS EKS
>>>
>>>
>>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 21:11
>>> tarihinde şunu yazdı:
>>>
>>>> I forgot top ask which k8s cluster are you using, assuming some clod
>>>> vendor
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <is...@gmail.com>
>>>> wrote:
>>>>
>>>>> and If you look at the code
>>>>>
>>>>>
>>>>> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
>>>>>
>>>>> .editOrNewResources()
>>>>> .addToRequests("memory", executorMemoryQuantity)
>>>>> .addToLimits("memory", executorMemoryQuantity)
>>>>> .addToRequests("cpu", executorCpuQuantity)
>>>>> .addToLimits(executorResourceQuantities.asJava)
>>>>> .endResources()
>>>>>
>>>>> addToRequests and addToLimits for memory have the same value.
>>>>> maybe it is by design. but can I set custom values for them if I use
>>>>> podtemplate?
>>>>>
>>>>>
>>>>>
>>>>> Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52
>>>>> tarihinde şunu yazdı:
>>>>>
>>>>>> Hi,
>>>>>> using spark version v.3.1.2
>>>>>>
>>>>>> spark.executor.memory is set.
>>>>>> But the problem is not setting spark.executor.memory, the problem is
>>>>>> that whatever value I set spark.executor.memory,
>>>>>> spark executor pod has the same value for resources.limit.memory and
>>>>>> resources.request.memory.
>>>>>> I want to be able to set different values for them.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
>>>>>> tarihinde şunu yazdı:
>>>>>>
>>>>>>> What are those currently set in spark-submit and which spark version
>>>>>>> on k8s
>>>>>>>
>>>>>>> --conf spark.driver.memory=2000m \
>>>>>>> --conf spark.executor.memory=2000m \
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> view my Linkedin profile
>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>>
>>>>>>>
>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>>> arise from relying on this email's technical content is explicitly
>>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>>> arising from such loss, damage or destruction.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <
>>>>>>> ismailyenigul@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> There is a cpu parameter to set spark executor on k8s
>>>>>>>> spark.kubernetes.executor.limit.cores and
>>>>>>>> spark.kubernetes.executor.request.cores
>>>>>>>> but there is no parameter to set memory request different then
>>>>>>>> limits memory (such as spark.kubernetes.executor.request.memory)
>>>>>>>> For that reason,
>>>>>>>> spark.executor.memory is assigned to requests.memory and
>>>>>>>> limits.memory like the following
>>>>>>>>
>>>>>>>> Limits:
>>>>>>>> memory: 5734MiRequests:
>>>>>>>> cpu: 4
>>>>>>>> memory: 5734Mi
>>>>>>>>
>>>>>>>>
>>>>>>>> Is there any special reason to not have
>>>>>>>> spark.kubernetes.executor.request.memory parameter?
>>>>>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter
>>>>>>>> to set smaller memory request than the memory limit in pod template file?
>>>>>>>>
>>>>>>>>
>>>>>>>> Limits:
>>>>>>>> memory: 5734MiRequests:
>>>>>>>> cpu: 4
>>>>>>>> memory: 1024Mi
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>>
Re: spark executor pod has same memory value for request and limit
Posted by Ismail Yenigul <is...@gmail.com>.
Hi Mich,
The issue is here there is no parameter to set executor pod request memory
value.
Currently we have only one parameter which is spark.executor.memory and it
set pod resources limit and requests.
Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 22:04
tarihinde şunu yazdı:
> Yes, both EKS and GKE (Google) are on 3.1.2 so I am not sure those
> parameters will work :(
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 10 Mar 2023 at 19:01, Ismail Yenigul <is...@gmail.com>
> wrote:
>
>> Hi Mich,
>>
>> it is on AWS EKS
>>
>>
>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 21:11
>> tarihinde şunu yazdı:
>>
>>> I forgot top ask which k8s cluster are you using, assuming some clod
>>> vendor
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <is...@gmail.com>
>>> wrote:
>>>
>>>> and If you look at the code
>>>>
>>>>
>>>> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
>>>>
>>>> .editOrNewResources()
>>>> .addToRequests("memory", executorMemoryQuantity)
>>>> .addToLimits("memory", executorMemoryQuantity)
>>>> .addToRequests("cpu", executorCpuQuantity)
>>>> .addToLimits(executorResourceQuantities.asJava)
>>>> .endResources()
>>>>
>>>> addToRequests and addToLimits for memory have the same value.
>>>> maybe it is by design. but can I set custom values for them if I use
>>>> podtemplate?
>>>>
>>>>
>>>>
>>>> Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52
>>>> tarihinde şunu yazdı:
>>>>
>>>>> Hi,
>>>>> using spark version v.3.1.2
>>>>>
>>>>> spark.executor.memory is set.
>>>>> But the problem is not setting spark.executor.memory, the problem is
>>>>> that whatever value I set spark.executor.memory,
>>>>> spark executor pod has the same value for resources.limit.memory and
>>>>> resources.request.memory.
>>>>> I want to be able to set different values for them.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
>>>>> tarihinde şunu yazdı:
>>>>>
>>>>>> What are those currently set in spark-submit and which spark version
>>>>>> on k8s
>>>>>>
>>>>>> --conf spark.driver.memory=2000m \
>>>>>> --conf spark.executor.memory=2000m \
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>>
>>>>>>
>>>>>> view my Linkedin profile
>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>>>
>>>>>>
>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility
>>>>>> for any loss, damage or destruction of data or any other property which may
>>>>>> arise from relying on this email's technical content is explicitly
>>>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>>>> arising from such loss, damage or destruction.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> There is a cpu parameter to set spark executor on k8s
>>>>>>> spark.kubernetes.executor.limit.cores and
>>>>>>> spark.kubernetes.executor.request.cores
>>>>>>> but there is no parameter to set memory request different then
>>>>>>> limits memory (such as spark.kubernetes.executor.request.memory)
>>>>>>> For that reason,
>>>>>>> spark.executor.memory is assigned to requests.memory and
>>>>>>> limits.memory like the following
>>>>>>>
>>>>>>> Limits:
>>>>>>> memory: 5734MiRequests:
>>>>>>> cpu: 4
>>>>>>> memory: 5734Mi
>>>>>>>
>>>>>>>
>>>>>>> Is there any special reason to not have
>>>>>>> spark.kubernetes.executor.request.memory parameter?
>>>>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to
>>>>>>> set smaller memory request than the memory limit in pod template file?
>>>>>>>
>>>>>>>
>>>>>>> Limits:
>>>>>>> memory: 5734MiRequests:
>>>>>>> cpu: 4
>>>>>>> memory: 1024Mi
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
Re: spark executor pod has same memory value for request and limit
Posted by Mich Talebzadeh <mi...@gmail.com>.
I forgot top ask which k8s cluster are you using, assuming some clod vendor
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 10 Mar 2023 at 18:00, Ismail Yenigul <is...@gmail.com>
wrote:
> and If you look at the code
>
>
> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
>
> .editOrNewResources()
> .addToRequests("memory", executorMemoryQuantity)
> .addToLimits("memory", executorMemoryQuantity)
> .addToRequests("cpu", executorCpuQuantity)
> .addToLimits(executorResourceQuantities.asJava)
> .endResources()
>
> addToRequests and addToLimits for memory have the same value.
> maybe it is by design. but can I set custom values for them if I use
> podtemplate?
>
>
>
> Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52
> tarihinde şunu yazdı:
>
>> Hi,
>> using spark version v.3.1.2
>>
>> spark.executor.memory is set.
>> But the problem is not setting spark.executor.memory, the problem is that
>> whatever value I set spark.executor.memory,
>> spark executor pod has the same value for resources.limit.memory and
>> resources.request.memory.
>> I want to be able to set different values for them.
>>
>>
>>
>>
>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
>> tarihinde şunu yazdı:
>>
>>> What are those currently set in spark-submit and which spark version on
>>> k8s
>>>
>>> --conf spark.driver.memory=2000m \
>>> --conf spark.executor.memory=2000m \
>>>
>>> HTH
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> There is a cpu parameter to set spark executor on k8s
>>>> spark.kubernetes.executor.limit.cores and
>>>> spark.kubernetes.executor.request.cores
>>>> but there is no parameter to set memory request different then limits
>>>> memory (such as spark.kubernetes.executor.request.memory)
>>>> For that reason,
>>>> spark.executor.memory is assigned to requests.memory and limits.memory
>>>> like the following
>>>>
>>>> Limits:
>>>> memory: 5734MiRequests:
>>>> cpu: 4
>>>> memory: 5734Mi
>>>>
>>>>
>>>> Is there any special reason to not have
>>>> spark.kubernetes.executor.request.memory parameter?
>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to
>>>> set smaller memory request than the memory limit in pod template file?
>>>>
>>>>
>>>> Limits:
>>>> memory: 5734MiRequests:
>>>> cpu: 4
>>>> memory: 1024Mi
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
Re: spark executor pod has same memory value for request and limit
Posted by Ismail Yenigul <is...@gmail.com>.
It means,
my requested feature is not available in the latest branch too ;)
Bjørn Jørgensen <bj...@gmail.com>, 10 Mar 2023 Cum, 21:05
tarihinde şunu yazdı:
> Strange to see that you are using spark 3.1.2 witch is EOL and you are
> reading source files from 3.4.0-SNAPSHOT
>
> fre. 10. mar. 2023 kl. 19:01 skrev Ismail Yenigul <ismailyenigul@gmail.com
> >:
>
>> and If you look at the code
>>
>>
>> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
>>
>> .editOrNewResources()
>> .addToRequests("memory", executorMemoryQuantity)
>> .addToLimits("memory", executorMemoryQuantity)
>> .addToRequests("cpu", executorCpuQuantity)
>> .addToLimits(executorResourceQuantities.asJava)
>> .endResources()
>>
>> addToRequests and addToLimits for memory have the same value.
>> maybe it is by design. but can I set custom values for them if I use
>> podtemplate?
>>
>>
>>
>> Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52
>> tarihinde şunu yazdı:
>>
>>> Hi,
>>> using spark version v.3.1.2
>>>
>>> spark.executor.memory is set.
>>> But the problem is not setting spark.executor.memory, the problem is
>>> that whatever value I set spark.executor.memory,
>>> spark executor pod has the same value for resources.limit.memory and
>>> resources.request.memory.
>>> I want to be able to set different values for them.
>>>
>>>
>>>
>>>
>>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
>>> tarihinde şunu yazdı:
>>>
>>>> What are those currently set in spark-submit and which spark version on
>>>> k8s
>>>>
>>>> --conf spark.driver.memory=2000m \
>>>> --conf spark.executor.memory=2000m \
>>>>
>>>> HTH
>>>>
>>>>
>>>>
>>>> view my Linkedin profile
>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>>
>>>>
>>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>>
>>>>
>>>>
>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>>> any loss, damage or destruction of data or any other property which may
>>>> arise from relying on this email's technical content is explicitly
>>>> disclaimed. The author will in no case be liable for any monetary damages
>>>> arising from such loss, damage or destruction.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> There is a cpu parameter to set spark executor on k8s
>>>>> spark.kubernetes.executor.limit.cores and
>>>>> spark.kubernetes.executor.request.cores
>>>>> but there is no parameter to set memory request different then limits
>>>>> memory (such as spark.kubernetes.executor.request.memory)
>>>>> For that reason,
>>>>> spark.executor.memory is assigned to requests.memory and
>>>>> limits.memory like the following
>>>>>
>>>>> Limits:
>>>>> memory: 5734MiRequests:
>>>>> cpu: 4
>>>>> memory: 5734Mi
>>>>>
>>>>>
>>>>> Is there any special reason to not have
>>>>> spark.kubernetes.executor.request.memory parameter?
>>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to
>>>>> set smaller memory request than the memory limit in pod template file?
>>>>>
>>>>>
>>>>> Limits:
>>>>> memory: 5734MiRequests:
>>>>> cpu: 4
>>>>> memory: 1024Mi
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: spark executor pod has same memory value for request and limit
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Strange to see that you are using spark 3.1.2 witch is EOL and you are
reading source files from 3.4.0-SNAPSHOT
fre. 10. mar. 2023 kl. 19:01 skrev Ismail Yenigul <is...@gmail.com>:
> and If you look at the code
>
>
> https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
>
> .editOrNewResources()
> .addToRequests("memory", executorMemoryQuantity)
> .addToLimits("memory", executorMemoryQuantity)
> .addToRequests("cpu", executorCpuQuantity)
> .addToLimits(executorResourceQuantities.asJava)
> .endResources()
>
> addToRequests and addToLimits for memory have the same value.
> maybe it is by design. but can I set custom values for them if I use
> podtemplate?
>
>
>
> Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52
> tarihinde şunu yazdı:
>
>> Hi,
>> using spark version v.3.1.2
>>
>> spark.executor.memory is set.
>> But the problem is not setting spark.executor.memory, the problem is that
>> whatever value I set spark.executor.memory,
>> spark executor pod has the same value for resources.limit.memory and
>> resources.request.memory.
>> I want to be able to set different values for them.
>>
>>
>>
>>
>> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
>> tarihinde şunu yazdı:
>>
>>> What are those currently set in spark-submit and which spark version on
>>> k8s
>>>
>>> --conf spark.driver.memory=2000m \
>>> --conf spark.executor.memory=2000m \
>>>
>>> HTH
>>>
>>>
>>>
>>> view my Linkedin profile
>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>>
>>>
>>> https://en.everybodywiki.com/Mich_Talebzadeh
>>>
>>>
>>>
>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>>> any loss, damage or destruction of data or any other property which may
>>> arise from relying on this email's technical content is explicitly
>>> disclaimed. The author will in no case be liable for any monetary damages
>>> arising from such loss, damage or destruction.
>>>
>>>
>>>
>>>
>>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> There is a cpu parameter to set spark executor on k8s
>>>> spark.kubernetes.executor.limit.cores and
>>>> spark.kubernetes.executor.request.cores
>>>> but there is no parameter to set memory request different then limits
>>>> memory (such as spark.kubernetes.executor.request.memory)
>>>> For that reason,
>>>> spark.executor.memory is assigned to requests.memory and limits.memory
>>>> like the following
>>>>
>>>> Limits:
>>>> memory: 5734MiRequests:
>>>> cpu: 4
>>>> memory: 5734Mi
>>>>
>>>>
>>>> Is there any special reason to not have
>>>> spark.kubernetes.executor.request.memory parameter?
>>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to
>>>> set smaller memory request than the memory limit in pod template file?
>>>>
>>>>
>>>> Limits:
>>>> memory: 5734MiRequests:
>>>> cpu: 4
>>>> memory: 1024Mi
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: spark executor pod has same memory value for request and limit
Posted by Ismail Yenigul <is...@gmail.com>.
and If you look at the code
https://github.com/apache/spark/blob/e64262f417bf381bdc664dfd1cbcfaa5aa7221fe/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L194
.editOrNewResources()
.addToRequests("memory", executorMemoryQuantity)
.addToLimits("memory", executorMemoryQuantity)
.addToRequests("cpu", executorCpuQuantity)
.addToLimits(executorResourceQuantities.asJava)
.endResources()
addToRequests and addToLimits for memory have the same value.
maybe it is by design. but can I set custom values for them if I use
podtemplate?
Ismail Yenigul <is...@gmail.com>, 10 Mar 2023 Cum, 20:52 tarihinde
şunu yazdı:
> Hi,
> using spark version v.3.1.2
>
> spark.executor.memory is set.
> But the problem is not setting spark.executor.memory, the problem is that
> whatever value I set spark.executor.memory,
> spark executor pod has the same value for resources.limit.memory and
> resources.request.memory.
> I want to be able to set different values for them.
>
>
>
>
> Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
> tarihinde şunu yazdı:
>
>> What are those currently set in spark-submit and which spark version on
>> k8s
>>
>> --conf spark.driver.memory=2000m \
>> --conf spark.executor.memory=2000m \
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> There is a cpu parameter to set spark executor on k8s
>>> spark.kubernetes.executor.limit.cores and
>>> spark.kubernetes.executor.request.cores
>>> but there is no parameter to set memory request different then limits
>>> memory (such as spark.kubernetes.executor.request.memory)
>>> For that reason,
>>> spark.executor.memory is assigned to requests.memory and limits.memory
>>> like the following
>>>
>>> Limits:
>>> memory: 5734MiRequests:
>>> cpu: 4
>>> memory: 5734Mi
>>>
>>>
>>> Is there any special reason to not have
>>> spark.kubernetes.executor.request.memory parameter?
>>> and can I use spark.kubernetes.executor.podTemplateFile parameter to set
>>> smaller memory request than the memory limit in pod template file?
>>>
>>>
>>> Limits:
>>> memory: 5734MiRequests:
>>> cpu: 4
>>> memory: 1024Mi
>>>
>>>
>>> Thanks
>>>
>>>
Re: spark executor pod has same memory value for request and limit
Posted by Ismail Yenigul <is...@gmail.com>.
Hi,
using spark version v.3.1.2
spark.executor.memory is set.
But the problem is not setting spark.executor.memory, the problem is that
whatever value I set spark.executor.memory,
spark executor pod has the same value for resources.limit.memory and
resources.request.memory.
I want to be able to set different values for them.
Mich Talebzadeh <mi...@gmail.com>, 10 Mar 2023 Cum, 20:44
tarihinde şunu yazdı:
> What are those currently set in spark-submit and which spark version on k8s
>
> --conf spark.driver.memory=2000m \
> --conf spark.executor.memory=2000m \
>
> HTH
>
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
> wrote:
>
>> Hi,
>>
>> There is a cpu parameter to set spark executor on k8s
>> spark.kubernetes.executor.limit.cores and
>> spark.kubernetes.executor.request.cores
>> but there is no parameter to set memory request different then limits
>> memory (such as spark.kubernetes.executor.request.memory)
>> For that reason,
>> spark.executor.memory is assigned to requests.memory and limits.memory
>> like the following
>>
>> Limits:
>> memory: 5734MiRequests:
>> cpu: 4
>> memory: 5734Mi
>>
>>
>> Is there any special reason to not have
>> spark.kubernetes.executor.request.memory parameter?
>> and can I use spark.kubernetes.executor.podTemplateFile parameter to set
>> smaller memory request than the memory limit in pod template file?
>>
>>
>> Limits:
>> memory: 5734MiRequests:
>> cpu: 4
>> memory: 1024Mi
>>
>>
>> Thanks
>>
>>
Re: spark executor pod has same memory value for request and limit
Posted by Mich Talebzadeh <mi...@gmail.com>.
What are those currently set in spark-submit and which spark version on k8s
--conf spark.driver.memory=2000m \
--conf spark.executor.memory=2000m \
HTH
view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
https://en.everybodywiki.com/Mich_Talebzadeh
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On Fri, 10 Mar 2023 at 17:39, Ismail Yenigul <is...@gmail.com>
wrote:
> Hi,
>
> There is a cpu parameter to set spark executor on k8s
> spark.kubernetes.executor.limit.cores and
> spark.kubernetes.executor.request.cores
> but there is no parameter to set memory request different then limits
> memory (such as spark.kubernetes.executor.request.memory)
> For that reason,
> spark.executor.memory is assigned to requests.memory and limits.memory
> like the following
>
> Limits:
> memory: 5734MiRequests:
> cpu: 4
> memory: 5734Mi
>
>
> Is there any special reason to not have
> spark.kubernetes.executor.request.memory parameter?
> and can I use spark.kubernetes.executor.podTemplateFile parameter to set
> smaller memory request than the memory limit in pod template file?
>
>
> Limits:
> memory: 5734MiRequests:
> cpu: 4
> memory: 1024Mi
>
>
> Thanks
>
>