You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Bigg Ben <ch...@gmail.com> on 2019/05/07 14:36:53 UTC

Running Spark 2.4 on K8S

Hi All,

I recently upgraded my spark version to 2.4.1 but when I try to run it with
the spark launcher 2.4, the set conf doesn't set my K8S environment
variables from the launcher submit command. Did anything change or am I
doing something wrong?

Thanks and Regards,

Re: Running Spark 2.4 on K8S

Posted by Bigg Ben <ch...@gmail.com>.

Just to provide more context. This is how the same spark launcher would
send my env variable for spark 2.3. Attached below... Those values are
missing when I change the versions of spark launcher and spark to 2.4. Any
one using spark 2.4 and spark launcher to submit spark applications?

On Tue, May 7, 2019, 10:36 Bigg Ben <ch...@gmail.com> wrote:

> Hi All,
>
> I recently upgraded my spark version to 2.4.1 but when I try to run it
> with the spark launcher 2.4, the set conf doesn't set my K8S environment
> variables from the launcher submit command. Did anything change or am I
> doing something wrong?
>
> Thanks and Regards,
>