You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mdagost <gi...@git.apache.org> on 2015/03/28 18:51:48 UTC
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
GitHub user mdagost opened a pull request:
https://github.com/apache/spark/pull/5244
SPARK-6588: Add --private-ips flag to EC2 script
The `spark_ec2.py` script currently references the `ip_address` and `public_dns_name` attributes of an instance. On private networks, these fields aren't set, so we have problems.
This PR introduces a `--private-ips` flag that instead refers to the `private_ip_address` attribute in both cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mdagost/spark ec2_private_nets
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5244.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5244
----
commit 2d9b7775ac1e6ae9942a2268567e176a4921ecf9
Author: Michelangelo D'Agostino <md...@civisanalytics.com>
Date: 2015-03-28T17:29:47Z
ENH: Add --private-ips flag.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90860193
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29853/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5244
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87275826
Ooops. Didn't realize this was a duplicate. I don't think I'm going to have time in the immediate future to make a general version, but the private IP stuff is a total blocker for us.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90945667
[Test build #29862 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29862/consoleFull) for PR 5244 at commit [`b684c67`](https://github.com/apache/spark/commit/b684c67fc622a6996a71d546de92e1a6ffbe6f49).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90156842
Sorry I missed some of the earlier discussion, but the change looks pretty simple and seems like a useful addition. Does this also address #4038 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-89281384
@nchammas Did you get a chance to take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87282950
@nchammas At the moment, we're using Spark for large scale ALS. In the future probably other things too. And we use spark-ec2 to spin up and tear down clusters as we need them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90920302
@srowen @nchammas Sorry about that. The lint checker should pass now.
One thing I forgot to mention yesterday: apparently someone got rid of the last place in the code where an instance was referred to by IP rather than DNS name. That means the helper function `get_ip_address` that I introduced isn't actually being called anywhere. I'd like to leave it in case someone needs IP in the future, but up to you guys.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90977078
Looks like @voukka closed #4038 in favor of this PR?
In any case, this patch LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by voukka <gi...@git.apache.org>.
Github user voukka commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-89505748
@srowen my 2 cents: I like https://issues.apache.org/jira/browse/SPARK-6220 as a general solution better as it is naturally extensible, but it is a lot less user-friendly. Probably they need to complement each other. For example, one can use --private-ips as parameter on cli, but for more options to pass one can use feature of https://issues.apache.org/jira/browse/SPARK-6220 and for cleaner cli feature of https://issues.apache.org/jira/browse/SPARK-925 could be used. I admit it looks like overkill. Still, I'd like to say that explicit parameters are easier to use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87278702
@nchammas @voukka what do you think about this as a solution for SPARK-5242 and/or SPARK-5246?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90859134
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87273976
You could take this all the way to implement what's described in SPARK-6220 as a more general version of this. And I think it would also be considered a resolution to SPARK-5242 and SPARK-5246. This overlaps with existing work at https://github.com/apache/spark/pull/4038 (see also https://github.com/mesos/spark-ec2/pull/91 and https://github.com/mesos/spark-ec2/pull/92) It'd be great to get a resolution to all of this since it keeps coming up in duplicate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90920951
[Test build #29862 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29862/consoleFull) for PR 5244 at commit [`b684c67`](https://github.com/apache/spark/commit/b684c67fc622a6996a71d546de92e1a6ffbe6f49).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87273816
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90859871
[Test build #29853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29853/consoleFull) for PR 5244 at commit [`a4a2eac`](https://github.com/apache/spark/commit/a4a2eaca3d67e5d59ca9c62b1375aa859d09aa17).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-87281232
I'll look into this next week, but @mdagost do you mind sharing what you are using spark-ec2 for? It's always good to hear about real-life use cases from users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90173226
> Does this also address #4038 ?
Yes, I believe it does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90009573
@mdagost how about addressing the minor comment typo and considering wrapping up the conditional logic -- up to you whether that is worthwhile. Otherwise looking good to go.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90945681
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29862/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90510047
Also, since this was a duplicate of SPARK-5242, edit the title please to point it at the non-duplicate JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90066630
Okay. I fixed the IP's mis-spelling and refactored the conditional logic into a couple of helper functions, which should be more readable now.
@nchammas I wrote this code and submitted the PR before I realized it was a dupe, so I never tried #4038 .
In terms of testing, I've verified that this works with our AWS setup to `launch` clusters into private VPC's, and it works with `get-master`, `login`, and `destroy`. I'm not entirely sure what you want me to do with `coverage`. This is the output that I get:
```
[ mdagostino: ~/spark_mdagost/ec2 ]$ coverage report spark_ec2.py
Name Stmts Miss Cover
-------------------------------
spark_ec2 677 509 25%
```
Do you actually want me to run the `launch`, `get-master`, `login`, and `destroy` commands and post the coverage reports for each of them?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by voukka <gi...@git.apache.org>.
Github user voukka commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-91434940
@nchammas that's correct, I closed #4038 in favor of this PR
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90117829
Oh, if you've tested out the relevant commands then that's great. I just wanted to know that we checked nothing was broken with this change. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-89439386
This is a well thought-out change. I prefer the explicit `--private-ips` option to the implicit address searching of #4038, but I must ask:
@mdagost Did you try the patch in #4038? Did it not meet your needs?
Also, while this change looks good, it touches a lot of code. Unfortunately, we don't have any automated test suites for spark-ec2 at this time.
In lieu of that, could you use [`coverage`](https://pypi.python.org/pypi/coverage/3.7.1) and post a report showing that you've tested most of the critical code paths?
Sorry to put this extra work on you, but that will help us merge this in with some confidence.
cc @shivaram
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90509835
@mdagost Do you mind rebase this one? it looks good but doesn't merge at the moment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by mdagost <gi...@git.apache.org>.
Github user mdagost commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90122010
Yep. I've been using it for the last week :)
On Mon, Apr 6, 2015 at 10:45 AM, Nicholas Chammas <no...@github.com>
wrote:
> Oh, if you've tested out the relevant commands then that's great. I just
> wanted to know that we checked nothing was broken with this change. :)
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/5244#issuecomment-90117829>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/5244#discussion_r27760408
--- Diff: ec2/spark_ec2.py ---
@@ -282,6 +282,10 @@ def parse_args():
parser.add_option(
"--vpc-id", default=None,
help="VPC to launch instances in")
+ parser.add_option(
+ "--private-ips", action="store_true", default=False,
+ help="Use private IP's for instances rather than public if VPC/subnet " +
--- End diff --
Nit: No possession or contraction, so "IPs" and not "IP's".
> Use private instead of public IPs for instances if the VPC or subnet requires that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-5242]: Add --private-ips flag to EC2 sc...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5244#issuecomment-90860182
[Test build #29853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29853/consoleFull) for PR 5244 at commit [`a4a2eac`](https://github.com/apache/spark/commit/a4a2eaca3d67e5d59ca9c62b1375aa859d09aa17).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: SPARK-6588: Add --private-ips flag to EC2 scri...
Posted by voukka <gi...@git.apache.org>.
Github user voukka commented on a diff in the pull request:
https://github.com/apache/spark/pull/5244#discussion_r27766535
--- Diff: ec2/spark_ec2.py ---
@@ -700,7 +704,8 @@ def get_instances(group_names):
# Deploy configuration files and run setup scripts on a newly launched
# or started EC2 cluster.
def setup_cluster(conn, master_nodes, slave_nodes, opts, deploy_ssh_key):
- master = master_nodes[0].public_dns_name
+ master = master_nodes[0].public_dns_name if not opts.private_ips else \
--- End diff --
Hello @mdagost! There's a conditional logic being repeated more than 10 times. Probably you might want to extract it in a function. @nchammas
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org