You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Li Gao <li...@gmail.com> on 2018/08/15 16:11:49 UTC

[K8S] Spark initContainer custom bootstrap support for Spark master

Hi,

We've noticed on the latest Master (not Spark 2.3.1 branch), the support
for Kubernetes initContainer is no longer there. What would be the path
forward if we need to do custom bootstrap actions (i.e. run additional
scripts) prior to driver/executor container entering running mode?

Thanks,
Li

Re: [K8S] Spark initContainer custom bootstrap support for Spark master

Posted by Li Gao <li...@gmail.com>.

Thanks! We will likely use the second option to customize the bootstrap.

On Thu, Aug 16, 2018 at 10:04 AM Yinan Li <li...@gmail.com> wrote:

> Yes, the init-container has been removed in the master branch. The
> init-container was used in 2.3.x only for downloading remote dependencies,
> which is now handled by running spark-submit in the driver. If you need to
> run custom bootstrap scripts using an init-container, the best option would
> be to use a mutating admission webhook to inject your init-container into
> the Spark pods. Another option is to create a custom image that runs the
> scripts prior to entering the entrypoint.
>
> Yinan
>
> On Wed, Aug 15, 2018 at 9:12 AM Li Gao <li...@gmail.com> wrote:
>
>> Hi,
>>
>> We've noticed on the latest Master (not Spark 2.3.1 branch), the support
>> for Kubernetes initContainer is no longer there. What would be the path
>> forward if we need to do custom bootstrap actions (i.e. run additional
>> scripts) prior to driver/executor container entering running mode?
>>
>> Thanks,
>> Li
>>
>>

Re: [K8S] Spark initContainer custom bootstrap support for Spark master

Posted by Yinan Li <li...@gmail.com>.

Yes, the init-container has been removed in the master branch. The
init-container was used in 2.3.x only for downloading remote dependencies,
which is now handled by running spark-submit in the driver. If you need to
run custom bootstrap scripts using an init-container, the best option would
be to use a mutating admission webhook to inject your init-container into
the Spark pods. Another option is to create a custom image that runs the
scripts prior to entering the entrypoint.

Yinan

On Wed, Aug 15, 2018 at 9:12 AM Li Gao <li...@gmail.com> wrote:

> Hi,
>
> We've noticed on the latest Master (not Spark 2.3.1 branch), the support
> for Kubernetes initContainer is no longer there. What would be the path
> forward if we need to do custom bootstrap actions (i.e. run additional
> scripts) prior to driver/executor container entering running mode?
>
> Thanks,
> Li
>
>