You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by douglaz <gi...@git.apache.org> on 2014/05/01 18:14:06 UTC
[GitHub] spark pull request: EC2 configurable workers
GitHub user douglaz opened a pull request:
https://github.com/apache/spark/pull/612
EC2 configurable workers
Added option to configure number of worker instances and to set SPARK_MASTER_OPTS
Depends on: https://github.com/mesos/spark-ec2/pull/46
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/douglaz/spark ec2_configurable_workers
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/612.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #612
----
commit ba528b911718675becf1623c803489b4cc3e68ec
Author: Allan Douglas R. de Oliveira <al...@chaordicsystems.com>
Date: 2014-04-14T00:09:47Z
Added SPARK_WORKER_INSTANCES parameter
commit 6c346716e5dd85bb4806ea2a71fa3a2fd00bbfcc
Author: Allan Douglas R. de Oliveira <al...@chaordicsystems.com>
Date: 2014-04-14T13:35:03Z
Use number of worker instances as string on template
commit d6c5d6564b3025eef88d499b253e850dd07f397a
Author: Allan Douglas R. de Oliveira <al...@chaordicsystems.com>
Date: 2014-05-01T14:50:14Z
Added master opts parameter
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/612
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-41925696
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-42119736
I merged this. I also opened up a new JIRA to just allow us to launch multiple executors for a given app on the same node:
https://issues.apache.org/jira/browse/SPARK-1706
That will provide a better general solution for users wanting to do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by douglaz <gi...@git.apache.org>.
Github user douglaz commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-42117075
Done:
https://issues.apache.org/jira/browse/SPARK-1705
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by douglaz <gi...@git.apache.org>.
Github user douglaz commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-42121342
Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-42119635
Thanks - this is a nice feature. I played with this locally and it worked.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: EC2 configurable workers
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/612#issuecomment-42114608
Do you mind creating a JIRA for this on the spark issue tracker?
https://issues.apache.org/jira/browse/SPARK
It would be good to call it something like: "Allow multiple instances per node with SPARK-EC2"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---