You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sigmoidanalytics <gi...@git.apache.org> on 2014/05/07 17:42:25 UTC
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
GitHub user sigmoidanalytics opened a pull request:
https://github.com/apache/spark/pull/681
Added SparkGCE Script for Version 0.9.1
I have added the SparkGCE script in this pull request. Just like the spark_ec2 script, this one also reads certain command-line arguments (See the README.md for more details) like the cluster name and all, then starts the machines in the google cloud, sets up the network, adds a 500GB empty disk to all machines, generate the ssh keys on master and transfer it to all slaves and install java and downloads and configures Spark-v0.9.1/Shark-v0.9.1/Hadoop-v0.9.1. Also it starts the shark server automatically. Currently the version is 0.9.1 but I'm happy to add/support more versions
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sigmoidanalytics/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/681.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #681
----
commit fc228d0bb45b5ecfbc1099b1b2bc9fe0cc3c4855
Author: AkhlD <ak...@darktech.ca>
Date: 2014-05-07T15:34:26Z
Added SparkGCE Script for Version 0.9.1
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-70395313
This is being maintained in it's own package now, so let's close this issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by sigmoidanalytics <gi...@git.apache.org>.
Github user sigmoidanalytics commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-63060313
LibCloud looks decent for launching machine across datacenters providers,
but to modify build scripts to either do installation from scratch (means
longer install times than spark ec2) or keep slightly baked
AMI's/equivalents in each datacenter of each provider. Both are sizable
amount of effort.
If the purpose of script is for development support then we can choose some
locations & start there before going all out. Another possibility is to
keep this all-datacenter script separate with longer launch times on vanila
AMI for those who want to use other datacenters apart from those supported
by spark-ec2. That way general launch times dont increase & still a
solution for those who dont want to use AWS.
On Fri, Nov 14, 2014 at 4:39 PM, Andrew Ash <no...@github.com>
wrote:
> Correction, please use SPARK-1422
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/681#issuecomment-63045363>.
>
--
Regards,
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <http://www.twitter.com/mayur_rustagi>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by sigmoidanalytics <gi...@git.apache.org>.
Github user sigmoidanalytics commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-42833253
Did any of the admin had chance to check it out? Let me know if you want me to modify anything in it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-50441094
Libcloud looks good actually, and it's nice that it's another Apache project. Would be worth a try if you guys want to investigate it. It would be awesome if we also get OpenStack as a result of it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-42443819
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-63024468
@sigmoidanalytics Did you ever take a look at libcloud to see if we could abstract out some commonality with the spark_ec2 scripts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-63045363
Correction, please use SPARK-1422
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by ash211 <gi...@git.apache.org>.
Github user ash211 commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-63043646
Oh and also, all GitHub PRs should have a Jira SPARK-XYZ title in the title. I created one for you -- can you please update this PR name to be "SPARK-4400 Added SparkGCE Script"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/681
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-54694717
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by mateiz <gi...@git.apache.org>.
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-50429284
@sigmoidanalytics sorry for the late reply on this, it looks like this got created during the 1.0 release period and reviewers missed it. Hopefully we can get this in 1.2. The main question I have is whether it's possible to share more code between this and the EC2 scripts. Is there any Python library that abstracts over the cloud provider?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: Added SparkGCE Script for Version 0.9.1
Posted by sigmoidanalytics <gi...@git.apache.org>.
Github user sigmoidanalytics commented on the pull request:
https://github.com/apache/spark/pull/681#issuecomment-50430585
Mostly https://libcloud.apache.org/ should work to abstract away some
launch code. Can look more deeply to check stability. Has any body used it
in some other project yet?
Whirr does some cloud level abstraction but would probably not apply here
as we are mostly deploying Spark only using this script.
Regards
Mayur
On Mon, Jul 28, 2014 at 8:00 PM, Matei Zaharia <no...@github.com>
wrote:
> @sigmoidanalytics <https://github.com/sigmoidanalytics> sorry for the
> late reply on this, it looks like this got created during the 1.0 release
> period and reviewers missed it. Hopefully we can get this in 1.2. The main
> question I have is whether it's possible to share more code between this
> and the EC2 scripts. Is there any Python library that abstracts over the
> cloud provider?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/681#issuecomment-50429284>.
>
--
Regards,
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <http://www.twitter.com/mayur_rustagi>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---