You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by huafengw <gi...@git.apache.org> on 2018/10/15 06:20:53 UTC
[GitHub] spark pull request #22722: [SPARK-24432][k8s] Add support for dynamic resour...
GitHub user huafengw opened a pull request:
https://github.com/apache/spark/pull/22722
[SPARK-24432][k8s] Add support for dynamic resource allocation on Kubernetes
## What changes were proposed in this pull request?
Enable dynamic resource allocation on Kubernetes by introducing an external shuffle servie on K8S.
## Acknowledgement
The basic idea and most of the code comes from project https://github.com/apache-spark-on-k8s/spark.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/huafengw/spark SPARK-24432
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22722.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22722
----
commit 4adcddb6177e5a79975e27f72915dfe18d6b7b2e
Author: huafengw <fv...@...>
Date: 2018-10-15T03:46:09Z
[SPARK-24432][k8s] Add support for dynamic resource allocation on Kubernetes
commit c2ba52bbb000291010580c8cb21f3254584190bc
Author: huafengw <fv...@...>
Date: 2018-10-15T06:08:33Z
Fix style
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by mccheah <gi...@git.apache.org>.
Github user mccheah commented on the issue:
https://github.com/apache/spark/pull/22722
Yep, please see https://issues.apache.org/jira/browse/SPARK-25299
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by ifilonenko <gi...@git.apache.org>.
Github user ifilonenko commented on the issue:
https://github.com/apache/spark/pull/22722
@huafengw Right now I am testing and further developing the experimental design (Option No. 5) that was proposed in https://issues.apache.org/jira/browse/SPARK-25299, building off of the work done by @mccheah. This is targetted for 3.0. I will throw up a WIP soon, which I would love your input on, once we convince the community on the design proposed in the WIP.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/spark/pull/22722
Hi, @liyinan926 can you help to review this pr?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by liyinan926 <gi...@git.apache.org>.
Github user liyinan926 commented on the issue:
https://github.com/apache/spark/pull/22722
@huafengw we are working on a new shuffle service + dynamic resource allocation design for k8s, and decided to not adopt the approach we used in the fork. @mccheah.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22722
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22722: [SPARK-24432][k8s] Add support for dynamic resource allo...
Posted by huafengw <gi...@git.apache.org>.
Github user huafengw commented on the issue:
https://github.com/apache/spark/pull/22722
Hi, @liyinan926 @mccheah, thanks for your reminding. Looks like it's quite a new design and needs a lot of work so what is the status right now? And is it planned to be in version 3.0?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org