You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by nchammas <gi...@git.apache.org> on 2014/11/11 01:12:25 UTC
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
GitHub user nchammas opened a pull request:
https://github.com/apache/spark/pull/3195
[SPARK-3398] [SPARK-4325] [EC2] Use EC2 status checks.
This PR re-introduces [0e648bc](https://github.com/apache/spark/commit/0e648bc2bedcbeb55fce5efac04f6dbad9f063b4) from PR #2339, which somehow never made it into the codebase.
Additionally, it removes a now-unnecessary linear backoff on the SSH checks since we are blocking on EC2 status checks before testing SSH.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nchammas/spark remove-ec2-ssh-backoff
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3195.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3195
----
commit 55caa24f39e21916d5c5c5e499ec3cd2dcd63392
Author: Nicholas Chammas <ni...@gmail.com>
Date: 2014-11-10T23:48:36Z
Check EC2 status checks before SSH.
commit adb4eaa4c44b0bcb3b6735c901f0750240dc0f7c
Author: Nicholas Chammas <ni...@gmail.com>
Date: 2014-11-10T23:54:12Z
Remove linear backoff.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63541152
@shivaram I think this PR is good to go. Did my response to your questions make sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-62488781
[Test build #23175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23175/consoleFull) for PR 3195 at commit [`adb4eaa`](https://github.com/apache/spark/commit/adb4eaa4c44b0bcb3b6735c901f0750240dc0f7c).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64702538
This seems okay to me, I guess. If we're being really conservative, I guess we could just leave the linear back-off in place.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-62481046
[Test build #23175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23175/consoleFull) for PR 3195 at commit [`adb4eaa`](https://github.com/apache/spark/commit/adb4eaa4c44b0bcb3b6735c901f0750240dc0f7c).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63522451
@nchammas I am bit confused about this change. The EC2 instance status checks seemed to be working fine when I tried things out last week, so I am not sure what we are missing. Is it just removing the exponential back-off ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-62612650
cc @JoshRosen @shivaram
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-66893722
Hmm, looks like I didn't resolve the JIRA issue here. Should this also be targeted for any backports, or is it fine to just have this in 1.3.0?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64939751
[Test build #23945 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23945/consoleFull) for PR 3195 at commit [`efb29e1`](https://github.com/apache/spark/commit/efb29e1799f2750da1787dc7256e1bcdd4db35fc).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-68024562
FYI: The commit message at 317e114 now incorrectly states that the linear backoff was removed. Is there any way to fix that?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3195
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63540834
[Test build #23556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23556/consoleFull) for PR 3195 at commit [`ef3ca99`](https://github.com/apache/spark/commit/ef3ca9966c0ac83a14ef1fae17506782405167fc).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/3195#discussion_r20526833
--- Diff: ec2/spark_ec2.py ---
@@ -655,33 +663,44 @@ def wait_for_cluster_state(cluster_instances, cluster_state, opts):
(would be nice to replace this with a proper enum: http://stackoverflow.com/a/1695250)
"""
sys.stdout.write(
- "Waiting for all instances in cluster to enter '{s}' state.".format(s=cluster_state)
+ "Waiting for cluster to enter '{s}' state.".format(s=cluster_state)
)
sys.stdout.flush()
- num_attempts = 0
+ start_time = datetime.now()
+
+ conn = ec2.connect_to_region(opts.region)
while True:
- time.sleep(3 * num_attempts)
+ time.sleep(5) # seconds
for i in cluster_instances:
- s = i.update() # capture output to suppress print to screen in newer versions of boto
--- End diff --
Is capturing the output not required anymore ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-62488790
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23175/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64825695
I guess I'd prefer to leave it in, but that's only because I don't want to risk a regression and don't really have time to figure out whether removing it is safe.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63527455
[Test build #23556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23556/consoleFull) for PR 3195 at commit [`ef3ca99`](https://github.com/apache/spark/commit/ef3ca9966c0ac83a14ef1fae17506782405167fc).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64945334
LGTM, so I'm going to merge this into `master`. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64938230
[Test build #23945 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23945/consoleFull) for PR 3195 at commit [`efb29e1`](https://github.com/apache/spark/commit/efb29e1799f2750da1787dc7256e1bcdd4db35fc).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3195#discussion_r20528480
--- Diff: ec2/spark_ec2.py ---
@@ -655,33 +663,44 @@ def wait_for_cluster_state(cluster_instances, cluster_state, opts):
(would be nice to replace this with a proper enum: http://stackoverflow.com/a/1695250)
"""
sys.stdout.write(
- "Waiting for all instances in cluster to enter '{s}' state.".format(s=cluster_state)
+ "Waiting for cluster to enter '{s}' state.".format(s=cluster_state)
)
sys.stdout.flush()
- num_attempts = 0
+ start_time = datetime.now()
+
+ conn = ec2.connect_to_region(opts.region)
--- End diff --
Good point. I'll fix that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63516554
Pinging @JoshRosen and @shivaram again.
This PR re-introduces a commit that somehow disappeared from #2339, and I think it should be good for inclusion in 1.2.0, since that's what it was originally slated for.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-68074192
Doh! Unfortunately, there's no way for us to go back and edit commit messages without messing up the git history. In the future, I'll be more careful when verifying commit message before merging (I should make a review checklist for these sorts of steps...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64711993
Yeah, I removed it specifically to be more aggressive and shave off some
seconds from the launch time. Do you think that's OK? Or would you prefer
the back off stay in?
On 2014년 11월 26일 (수) at 오후 2:59 Josh Rosen <no...@github.com> wrote:
> This seems okay to me, I guess. If we're being really conservative, I
> guess we could just leave the linear back-off in place.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/3195#issuecomment-64702538>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by JoshRosen <gi...@git.apache.org>.
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-66896346
Alright, in that case I'm going to mark SPARK-4325 as 'Resolved'. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64938212
OK @JoshRosen, I've added the backoff back in.
I didn't notice any regressions without the linear backoff, so I'll make a note to remove it in a future version and document the time savings. Fair enough?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-64939752
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23945/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63551750
Yeah this makes sense, thanks for clarifying.
One thing I am still thinking about is if it makes sense to do constant poll vs. linear back-off. From what I can see we make the EC2 status checks now every 5 seconds. I think that should be fine, though in the past I have run into cases where EC2 requests get rate-limited and start failing.
@JoshRosen any other thoughts ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63524432
`spark-ec2` currently blocks on SSH availability correctly, but it tests SSH more often than is necessary. As [you suggested here](https://github.com/apache/spark/pull/2339#issuecomment-55546927), this PR [reintroduces](https://github.com/apache/spark/pull/3195/files#diff-ada66bbeb2f1327b508232ef6c3805a5R680) a change that makes `spark-ec2` check the EC2 status checks _before_ trying SSH. This change was already in #2339, but someone got dropped. I'm not sure how.
Remember we were [concerned](https://github.com/apache/spark/pull/2339#issuecomment-55481878) about forking too many processes to test SSH. Without this patch, `spark-ec2` tests SSH the moment the instances are marked as online. With this patch, `spark-ec2` waits until the EC2 status checks additionally clear before testing SSH.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3195#discussion_r20529472
--- Diff: ec2/spark_ec2.py ---
@@ -655,33 +663,44 @@ def wait_for_cluster_state(cluster_instances, cluster_state, opts):
(would be nice to replace this with a proper enum: http://stackoverflow.com/a/1695250)
"""
sys.stdout.write(
- "Waiting for all instances in cluster to enter '{s}' state.".format(s=cluster_state)
+ "Waiting for cluster to enter '{s}' state.".format(s=cluster_state)
)
sys.stdout.flush()
- num_attempts = 0
+ start_time = datetime.now()
+
+ conn = ec2.connect_to_region(opts.region)
--- End diff --
Fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on a diff in the pull request:
https://github.com/apache/spark/pull/3195#discussion_r20526916
--- Diff: ec2/spark_ec2.py ---
@@ -655,33 +663,44 @@ def wait_for_cluster_state(cluster_instances, cluster_state, opts):
(would be nice to replace this with a proper enum: http://stackoverflow.com/a/1695250)
"""
sys.stdout.write(
- "Waiting for all instances in cluster to enter '{s}' state.".format(s=cluster_state)
+ "Waiting for cluster to enter '{s}' state.".format(s=cluster_state)
)
sys.stdout.flush()
- num_attempts = 0
+ start_time = datetime.now()
+
+ conn = ec2.connect_to_region(opts.region)
--- End diff --
We seem to be passing around `conn` across functions in the existing script. Any reason to not do that and create a new one here ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-66895881
@JoshRosen This work was originally supposed to be in for 1.2.0, but since it's not a critical piece of work I don't think there is a need to backport it for 1.2.1 or whatnot.
All it does is reduce the number of SSH calls made during launch by using the EC2 status checks, so I think it's OK to leave it for 1.3.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3195#issuecomment-63540846
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23556/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-3398] [SPARK-4325] [EC2] Use EC2 status...
Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:
https://github.com/apache/spark/pull/3195#discussion_r20527728
--- Diff: ec2/spark_ec2.py ---
@@ -655,33 +663,44 @@ def wait_for_cluster_state(cluster_instances, cluster_state, opts):
(would be nice to replace this with a proper enum: http://stackoverflow.com/a/1695250)
"""
sys.stdout.write(
- "Waiting for all instances in cluster to enter '{s}' state.".format(s=cluster_state)
+ "Waiting for cluster to enter '{s}' state.".format(s=cluster_state)
)
sys.stdout.flush()
- num_attempts = 0
+ start_time = datetime.now()
+
+ conn = ec2.connect_to_region(opts.region)
while True:
- time.sleep(3 * num_attempts)
+ time.sleep(5) # seconds
for i in cluster_instances:
- s = i.update() # capture output to suppress print to screen in newer versions of boto
--- End diff --
Nope, it never was.
When I originally added this in, I was testing in the Python shell and got confused. `i.update()` echoes its return value to the shell, so I thought I needed to suppress that output. But as part of a program, nothing gets echoed. We don't need the return value, so there's no need to capture it.
It's like just typing `var` in the shell and seeing its value. As part of a program, `var` alone will do nothing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org