You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Simon van der Veldt <si...@gmail.com> on 2022/04/07 19:37:55 UTC
Re: Reason/rationale for using a service for the driver when running Spark on Kubernetes?
Hi Mitch, Spark devs,
Thanks for the additional info and links. The PR you linked to had some
additional links and via one of them I found this issue
https://issues.apache.org/jira/browse/SPARK-21642
It seems that HTTPS/SSL support for the driver/Spark UI, which (generally)
requires a FQDN, was the reason for adding the Kubernetes service.
I understand that HTTPS/SSL might be relevant for a long running Spark
service/UI that one wants to safely expose, but at least in my experience
that's an exceptional case and not the normal one.
For example at my current project we run all our Spark workloads in their
own container/pods and do not even expose the driver's port let alone want
to make the Spark UI for all those drivers only work over HTTPS, so having
all these services be created/lifecycled seems like a lot of unnecessary
overhead.
We do also have long running Spark services but these are in the minority,
like say 1 out of 250 or so. And for most of those the Spark UI isn't
exposed either.
Would it maybe be possible to have some way of opting out of creating the
service?
Thanks again for your help!
Simon
On Thu, Mar 3, 2022 at 5:38 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> I looked at this document in more detail.
> https://github.com/apache/spark/pull/19717
>
> Quote
> "This PR contains implementation of the basic submission client for the
> cluster mode of Spark on Kubernetes. It's step 2 from the step-wise plan
> documented here
> <https://github.com/apache-spark-on-k8s/spark/issues/441#issuecomment-330802935>
> .
>
> This addition is covered by the SPIP
> <http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-td22147.html> vote
> which passed on Aug 31.
>
> This PR and #19468 <https://github.com/apache/spark/pull/19468> together
> form a MVP of Spark on Kubernetes that allows users to run Spark
> applications that use resources locally within the driver and executor
> containers on Kubernetes 1.6 and up. Some changes on pom and build/test
> setup are copied over from #19468
> <https://github.com/apache/spark/pull/19468> to make this PR self
> contained and testable.
>
> The submission client is mainly responsible for creating the Kubernetes
> pod that runs the Spark driver. It follows a step-based approach to
> construct the driver pod, as the code under the submit.steps package
> shows. The steps are orchestrated by DriverConfigurationStepsOrchestrator
> . Client creates the driver pod and waits for the application to complete
> if it's configured to do so, which is the case by default.
>
> This PR also contains Dockerfiles of the driver and executor images. They
> are included because some of the environment variables set in the code
> would not make sense without referring to the Dockerfiles."
>
> So far so good there are no surprises here. You expect in containerised
> applications one to create services. The point is that the driver service
> needs to be created first.
>
>
> [image: image.png]
>
>
>
> 1. As labelled in the diagram above, processes 1-3 will be dealing
> with the request for Kube apiserver to schedule the creation of the driver
> pod as a service and at the completion, process 4 originating from the
> driver will request kube apiserver to schedule creation of executors as
> services.
> 2. Depending on the number of executors requested through
> spark.executor.instances property, the driver will ask for these in chunks
> i.e. it will ask for N executors, wait briefly and then ask for another N
> executors. Where N is a controllable batch size (default 5) managed via the
> spark.kubernetes.allocation.batch.size property and the delay between
> batches of executors (default 1s) controlled by the
> spark.kubernetes.allocation.batch.delay property. This batch based
> allocation is done to avoid overwhelming the Kube apiserver and effectively
> overloading the cluster by asking for all the pods at once.
> 3. Processes 5 and 6 will be dealing with creating executors through
> the scheduler.
>
>
> In short, Kubernetes https://kubernetes.io/, is an open-source system for
> automating deployment, scaling, and management of containerized
> applications. the notation of "... The submission client is mainly
> responsible for creating the Kubernetes pod that runs the Spark driver...",
> may be a bit vague. The client requests the Kube apiserver to schedule the
> creation of the driver pod first and henceforth the creation of executtors.
>
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 2 Mar 2022 at 22:48, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi Simon,
>>
>> These are relatively old documents. Specifically, what issues are you
>> foreseeing because of the driver service?
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Wed, 2 Mar 2022 at 21:42, Simon van der Veldt <
>> simon.vanderveldt@gmail.com> wrote:
>>
>>> Hi Spark devs,
>>>
>>> Does anyone know what the reason/rationale was for using/creating a
>>> service for the driver when running Spark on Kubernetes?
>>> The PR (https://github.com/apache/spark/pull/19717) and the commit
>>> (
>>> https://github.com/mccheah/spark/commit/3f4060c340d6bac412e8819c4388ccba226efcf3
>>> )
>>> that added the initial support already included the driver service but
>>> I couldn't find any explanation as to why in there. The SPIP (22147
>>> https://lists.apache.org/thread/54rp03ywplr8tsvs76zgdkoq6yztrlvv) also
>>> doesn't document anything about it.
>>>
>>> Thanks in advance!
>>> Simon
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
On Thu, Mar 3, 2022 at 5:38 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> I looked at this document in more detail.
> https://github.com/apache/spark/pull/19717
>
> Quote
> "This PR contains implementation of the basic submission client for the
> cluster mode of Spark on Kubernetes. It's step 2 from the step-wise plan
> documented here
> <https://github.com/apache-spark-on-k8s/spark/issues/441#issuecomment-330802935>
> .
>
> This addition is covered by the SPIP
> <http://apache-spark-developers-list.1001551.n3.nabble.com/SPIP-Spark-on-Kubernetes-td22147.html> vote
> which passed on Aug 31.
>
> This PR and #19468 <https://github.com/apache/spark/pull/19468> together
> form a MVP of Spark on Kubernetes that allows users to run Spark
> applications that use resources locally within the driver and executor
> containers on Kubernetes 1.6 and up. Some changes on pom and build/test
> setup are copied over from #19468
> <https://github.com/apache/spark/pull/19468> to make this PR self
> contained and testable.
>
> The submission client is mainly responsible for creating the Kubernetes
> pod that runs the Spark driver. It follows a step-based approach to
> construct the driver pod, as the code under the submit.steps package
> shows. The steps are orchestrated by DriverConfigurationStepsOrchestrator
> . Client creates the driver pod and waits for the application to complete
> if it's configured to do so, which is the case by default.
>
> This PR also contains Dockerfiles of the driver and executor images. They
> are included because some of the environment variables set in the code
> would not make sense without referring to the Dockerfiles."
>
> So far so good there are no surprises here. You expect in containerised
> applications one to create services. The point is that the driver service
> needs to be created first.
>
>
> [image: image.png]
>
>
>
> 1. As labelled in the diagram above, processes 1-3 will be dealing
> with the request for Kube apiserver to schedule the creation of the driver
> pod as a service and at the completion, process 4 originating from the
> driver will request kube apiserver to schedule creation of executors as
> services.
> 2. Depending on the number of executors requested through
> spark.executor.instances property, the driver will ask for these in chunks
> i.e. it will ask for N executors, wait briefly and then ask for another N
> executors. Where N is a controllable batch size (default 5) managed via the
> spark.kubernetes.allocation.batch.size property and the delay between
> batches of executors (default 1s) controlled by the
> spark.kubernetes.allocation.batch.delay property. This batch based
> allocation is done to avoid overwhelming the Kube apiserver and effectively
> overloading the cluster by asking for all the pods at once.
> 3. Processes 5 and 6 will be dealing with creating executors through
> the scheduler.
>
>
> In short, Kubernetes https://kubernetes.io/, is an open-source system for
> automating deployment, scaling, and management of containerized
> applications. the notation of "... The submission client is mainly
> responsible for creating the Kubernetes pod that runs the Spark driver...",
> may be a bit vague. The client requests the Kube apiserver to schedule the
> creation of the driver pod first and henceforth the creation of executtors.
>
>
> HTH
>
>
> view my Linkedin profile
> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>
>
> https://en.everybodywiki.com/Mich_Talebzadeh
>
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Wed, 2 Mar 2022 at 22:48, Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi Simon,
>>
>> These are relatively old documents. Specifically, what issues are you
>> foreseeing because of the driver service?
>>
>> HTH
>>
>>
>>
>> view my Linkedin profile
>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>
>>
>>
>> https://en.everybodywiki.com/Mich_Talebzadeh
>>
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>>
>> On Wed, 2 Mar 2022 at 21:42, Simon van der Veldt <
>> simon.vanderveldt@gmail.com> wrote:
>>
>>> Hi Spark devs,
>>>
>>> Does anyone know what the reason/rationale was for using/creating a
>>> service for the driver when running Spark on Kubernetes?
>>> The PR (https://github.com/apache/spark/pull/19717) and the commit
>>> (
>>> https://github.com/mccheah/spark/commit/3f4060c340d6bac412e8819c4388ccba226efcf3
>>> )
>>> that added the initial support already included the driver service but
>>> I couldn't find any explanation as to why in there. The SPIP (22147
>>> https://lists.apache.org/thread/54rp03ywplr8tsvs76zgdkoq6yztrlvv) also
>>> doesn't document anything about it.
>>>
>>> Thanks in advance!
>>> Simon
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>