You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Pradeepta Choudhury <pr...@gmail.com> on 2020/05/09 16:34:20 UTC
dynamic executor scalling spark on kubernetes client mode
Hiii ,
The dynamic executor scalling is working fine for spark on kubernetes
(latest from spark master repository ) in cluster mode . is the dynamic
executor scalling available for client mode ? if yes where can i find the
usage doc for same .
If no is there any PR open for this ?
Thanks ,
Pradeepta
Re: dynamic executor scalling spark on kubernetes client mode
Posted by Steven Stetzler <st...@gmail.com>.
Oh, thanks for mentioning that, it looks l dynamic allocation on Kubernetes
works in client mode in Spark 3.0.0. I just had to set the following
configurations:
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.shuffleTracking.enabled=true
to enable dynamic allocation and disable the need for the external shuffle
service (which looks like it is experimental right now). My executor pods
couldn't connect to the external shuffle service when it was enabled. This
seems to be working okay for me.
Thanks,
Steven
On Tue, May 12, 2020 at 4:42 AM Pradeepta Choudhury <
pradeeptachoudhury1@gmail.com> wrote:
> Hey guys i was able to run dynamic scaling in both cluster and client mode
> . would document and send it over this weekend
>
> On Tue 12 May, 2020, 1:26 PM Roland Johann, <ro...@phenetic.io>
> wrote:
>
>> Hi all,
>>
>> don’t want to interrupt the conversation but are keen where I can find
>> information regarding dynamic allocation on kubernetes. As far as I know
>> the docs just point to future work.
>>
>> Thanks a lot,
>> Roland
>>
>>
>>
>> Am 12.05.2020 um 09:25 schrieb Steven Stetzler <steven.stetzler@gmail.com
>> >:
>>
>> Hi all,
>>
>> I am interested in this as well. My use-case could benefit from dynamic
>> executor scaling but we are restricted to using client mode since we are
>> only using Spark shells.
>>
>> Could anyone help me understand the barriers to getting dynamic executor
>> scaling to work in client mode on Kubernetes?
>>
>> Thanks,
>> Steven
>>
>> On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <
>> pradeeptachoudhury1@gmail.com> wrote:
>>
>>> Hiii ,
>>>
>>> The dynamic executor scalling is working fine for spark on kubernetes
>>> (latest from spark master repository ) in cluster mode . is the dynamic
>>> executor scalling available for client mode ? if yes where can i find the
>>> usage doc for same .
>>> If no is there any PR open for this ?
>>>
>>> Thanks ,
>>> Pradeepta
>>>
>>
>>
Re: dynamic executor scalling spark on kubernetes client mode
Posted by Pradeepta Choudhury <pr...@gmail.com>.
Hey guys i was able to run dynamic scaling in both cluster and client mode
. would document and send it over this weekend
On Tue 12 May, 2020, 1:26 PM Roland Johann, <ro...@phenetic.io>
wrote:
> Hi all,
>
> don’t want to interrupt the conversation but are keen where I can find
> information regarding dynamic allocation on kubernetes. As far as I know
> the docs just point to future work.
>
> Thanks a lot,
> Roland
>
>
>
> Am 12.05.2020 um 09:25 schrieb Steven Stetzler <steven.stetzler@gmail.com
> >:
>
> Hi all,
>
> I am interested in this as well. My use-case could benefit from dynamic
> executor scaling but we are restricted to using client mode since we are
> only using Spark shells.
>
> Could anyone help me understand the barriers to getting dynamic executor
> scaling to work in client mode on Kubernetes?
>
> Thanks,
> Steven
>
> On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <
> pradeeptachoudhury1@gmail.com> wrote:
>
>> Hiii ,
>>
>> The dynamic executor scalling is working fine for spark on kubernetes
>> (latest from spark master repository ) in cluster mode . is the dynamic
>> executor scalling available for client mode ? if yes where can i find the
>> usage doc for same .
>> If no is there any PR open for this ?
>>
>> Thanks ,
>> Pradeepta
>>
>
>
Re: dynamic executor scalling spark on kubernetes client mode
Posted by Roland Johann <ro...@phenetic.io.INVALID>.
Hi all,
don’t want to interrupt the conversation but are keen where I can find information regarding dynamic allocation on kubernetes. As far as I know the docs just point to future work.
Thanks a lot,
Roland
> Am 12.05.2020 um 09:25 schrieb Steven Stetzler <st...@gmail.com>:
>
> Hi all,
>
> I am interested in this as well. My use-case could benefit from dynamic executor scaling but we are restricted to using client mode since we are only using Spark shells.
>
> Could anyone help me understand the barriers to getting dynamic executor scaling to work in client mode on Kubernetes?
>
> Thanks,
> Steven
>
> On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <pradeeptachoudhury1@gmail.com <ma...@gmail.com>> wrote:
> Hiii ,
>
> The dynamic executor scalling is working fine for spark on kubernetes (latest from spark master repository ) in cluster mode . is the dynamic executor scalling available for client mode ? if yes where can i find the usage doc for same .
> If no is there any PR open for this ?
>
> Thanks ,
> Pradeepta
Re: dynamic executor scalling spark on kubernetes client mode
Posted by Steven Stetzler <st...@gmail.com>.
Hi all,
I am interested in this as well. My use-case could benefit from dynamic
executor scaling but we are restricted to using client mode since we are
only using Spark shells.
Could anyone help me understand the barriers to getting dynamic executor
scaling to work in client mode on Kubernetes?
Thanks,
Steven
On Sat, May 9, 2020 at 9:48 AM Pradeepta Choudhury <
pradeeptachoudhury1@gmail.com> wrote:
> Hiii ,
>
> The dynamic executor scalling is working fine for spark on kubernetes
> (latest from spark master repository ) in cluster mode . is the dynamic
> executor scalling available for client mode ? if yes where can i find the
> usage doc for same .
> If no is there any PR open for this ?
>
> Thanks ,
> Pradeepta
>