You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by 曹礼俊 <ca...@gmail.com> on 2018/10/26 09:14:52 UTC

External shuffle service on K8S

Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes?

I have looked up the documentation(
https://spark.apache.org/docs/latest/running-on-kubernetes.html), but
couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao

RE: External shuffle service on K8S

Posted by "Garlapati, Suryanarayana (Nokia - IN/Bangalore)" <su...@nokia.com>.
Hi,
There is an unmerged PR which can be used against spark 2.4(if you are interested) or master branch(3.0). Spark 2.3 K8S lacks lot of features. I suggest you upgrade to 2.4 which will be released in few days from now.

https://github.com/apache/spark/pull/22722

Regards
Surya

From: Matt Cheah <mc...@palantir.com>
Sent: Saturday, October 27, 2018 6:12 AM
To: Li Gao <ey...@gmail.com>; vincent.gromakowski@gmail.com
Cc: caolijun1166@gmail.com; user@spark.apache.org
Subject: Re: External shuffle service on K8S

Hi there,

Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this matter.

-Matt Cheah

From: Li Gao <ey...@gmail.com>>
Date: Friday, October 26, 2018 at 9:10 AM
To: "vincent.gromakowski@gmail.com<ma...@gmail.com>" <vi...@gmail.com>>
Cc: "caolijun1166@gmail.com<ma...@gmail.com>" <ca...@gmail.com>>, "user@spark.apache.org<ma...@spark.apache.org>" <us...@spark.apache.org>>
Subject: Re: External shuffle service on K8S

There are existing 2.2 based ext shuffle on the fork:
https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html [apache-spark-on-k8s.github.io]<https://urldefense.proofpoint.com/v2/url?u=https-3A__apache-2Dspark-2Don-2Dk8s.github.io_userdocs_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=enaZSDsU2MW_A_HY6l9TvqhhqoKOMWjbPvEOngFz1oA&e=>

You can modify it to suit your needs.

-Li


On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <vi...@gmail.com>> wrote:
No it's on the roadmap >2.4

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <ca...@gmail.com>> a écrit :
Hi all:

Does Spark 2.3.2 supports external shuffle service on Kubernetes?

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]<https://urldefense.proofpoint.com/v2/url?u=https-3A__spark.apache.org_docs_latest_running-2Don-2Dkubernetes.html&d=DwMFaQ&c=izlc9mHr637UR4lpLEZLFFS3Vn2UXBrZ4tFb6oOnmz8&r=hzwIMNQ9E99EMYGuqHI0kXhVbvX3nU3OSDadUnJxjAs&m=gB40LFy6y_6gk7_ZrhJYSJx3zyZLmmvfIEQeZvVpLog&s=1uVGe6F4jUe4jMJnoKyMPlbr4qanZg4AL4vVWuwMjaY&e=>), but couldn't find related suggestions.

If suppports, how can I enable it?

Best Regards

Lijun Cao

Re: External shuffle service on K8S

Posted by Matt Cheah <mc...@palantir.com>.
Hi there,

 

Please see https://issues.apache.org/jira/browse/SPARK-25299 for more discussion around this matter.

 

-Matt Cheah

 

From: Li Gao <ey...@gmail.com>
Date: Friday, October 26, 2018 at 9:10 AM
To: "vincent.gromakowski@gmail.com" <vi...@gmail.com>
Cc: "caolijun1166@gmail.com" <ca...@gmail.com>, "user@spark.apache.org" <us...@spark.apache.org>
Subject: Re: External shuffle service on K8S

 

There are existing 2.2 based ext shuffle on the fork: 

https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html [apache-spark-on-k8s.github.io]

 

You can modify it to suit your needs.

 

-Li

 

 

On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <vi...@gmail.com> wrote:

No it's on the roadmap >2.4

 

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <ca...@gmail.com> a écrit :

Hi all: 

 

Does Spark 2.3.2 supports external shuffle service on Kubernetes? 

 

I have looked up the documentation(https://spark.apache.org/docs/latest/running-on-kubernetes.html [spark.apache.org]), but couldn't find related suggestions.

 

If suppports, how can I enable it?

 

Best Regards

 

Lijun Cao

Re: External shuffle service on K8S

Posted by Li Gao <ey...@gmail.com>.
There are existing 2.2 based ext shuffle on the fork:
https://apache-spark-on-k8s.github.io/userdocs/running-on-kubernetes.html

You can modify it to suit your needs.

-Li


On Fri, Oct 26, 2018 at 3:22 AM vincent gromakowski <
vincent.gromakowski@gmail.com> wrote:

> No it's on the roadmap >2.4
>
> Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <ca...@gmail.com> a écrit :
>
>> Hi all:
>>
>> Does Spark 2.3.2 supports external shuffle service on Kubernetes?
>>
>> I have looked up the documentation(
>> https://spark.apache.org/docs/latest/running-on-kubernetes.html), but
>> couldn't find related suggestions.
>>
>> If suppports, how can I enable it?
>>
>> Best Regards
>>
>> Lijun Cao
>>
>>
>>
>

Re: External shuffle service on K8S

Posted by vincent gromakowski <vi...@gmail.com>.
No it's on the roadmap >2.4

Le ven. 26 oct. 2018 à 11:15, 曹礼俊 <ca...@gmail.com> a écrit :

> Hi all:
>
> Does Spark 2.3.2 supports external shuffle service on Kubernetes?
>
> I have looked up the documentation(
> https://spark.apache.org/docs/latest/running-on-kubernetes.html), but
> couldn't find related suggestions.
>
> If suppports, how can I enable it?
>
> Best Regards
>
> Lijun Cao
>
>
>