You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanji84 <gi...@git.apache.org> on 2017/05/15 01:38:01 UTC
[GitHub] spark pull request #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard ...
GitHub user yanji84 opened a pull request:
https://github.com/apache/spark/pull/17979
[SPARK-19320][MESOS][WIP]allow specifying a hard limit on number of gpus required in each spark executor when running on mesos
## What changes were proposed in this pull request?
Currently, Spark only allows specifying overall gpu resources as an upper limit, this adds a new conf parameter to allow specifying a hard limit on the number of gpu cores for each executor while still respecting the overall gpu resource constraint
## How was this patch tested?
Unit testing
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanji84/spark ji/set_allow_set_docker_user
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17979.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17979
----
commit 5f8ccd5789137363e035d1dfb9a05d3b9bf3ce6b
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-03-10T05:30:11Z
respect both gpu and maxgpu
commit 33ebff693d9b78a15221f931dbbca777cba944e0
Author: Ji Yan <ya...@gmail.com>
Date: 2017-03-10T05:43:21Z
Merge branch 'master' into ji/hard_limit_on_gpu
commit c2c1c5b66436a439e1d7342b7a2c58c502e26d6b
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-03-10T05:30:11Z
respect both gpu and maxgpu
commit c5c5c379fc27f579952700fdf2d15dae9eba104a
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-05-13T16:25:48Z
Merge branch 'ji/hard_limit_on_gpu' of https://github.com/yanji84/spark into ji/hard_limit_on_gpu
commit ba87b35817a7288b9b6aa41f4ac2244e235f2efd
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-05-13T16:53:59Z
fix syntax
commit 5ef2881a2b1e1180b73d532988bab72c5fdab64c
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-05-14T20:02:16Z
fix gpu offer
commit c301f3d1e05cc7359142a6cfb8222ad65cbdd9eb
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-05-14T20:15:55Z
syntax fix
commit 7a07742f4e004e0e88aa2b3bc5143adab3689644
Author: Ji Yan <ji...@jis-macbook-air.local>
Date: 2017-05-15T00:30:50Z
pass all tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by susanxhuynh <gi...@git.apache.org>.
Github user susanxhuynh commented on the issue:
https://github.com/apache/spark/pull/17979
I'm interested in this PR, too. Who has permission to reopen this? cc @HyukjinKwon @yanji84
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by grisaitis <gi...@git.apache.org>.
Github user grisaitis commented on the issue:
https://github.com/apache/spark/pull/17979
@yanji84 how is this different from your other PR, https://github.com/apache/spark/pull/17235?
I'm really interested in using this. I'll try testing it out on a GPU-enabled mesos cluster in the coming week.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/17979
@yanji84 you are the author, do you see the option to reopen this PR in github.com?
if not, feel free to open a new PR and @ me there
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17979
@yanji84, is this PR active? if so, would you answer to the question above?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17979
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17979
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by yanji84 <gi...@git.apache.org>.
Github user yanji84 commented on the issue:
https://github.com/apache/spark/pull/17979
We are really in need of this. Can we reopen this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17979: [SPARK-19320][MESOS][WIP]allow specifying a hard limit o...
Posted by kaluzniacki <gi...@git.apache.org>.
Github user kaluzniacki commented on the issue:
https://github.com/apache/spark/pull/17979
I am looking for a way to assign one GPU per RDD partition. I'll look at this patch in more detail, but a what I am fighting now is all GPUs on one executor are getting assigned to a single Mesos task rather than 1 GPU per Mesos task, and I think Mesos task maps to partition.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org