You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by mrdrozdov <gi...@git.apache.org> on 2015/03/24 21:16:48 UTC

[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

GitHub user mrdrozdov opened a pull request:

    https://github.com/apache/spark/pull/5172

    [SPARK-6474][EC2] Use EC2Connection.run_instances to launch clusters

    Instead of using the following call in spark_ec2.py to launch clusters:
    
    - [`ec2.image.Image.run`](http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.image.Image.run)
    
    this patch uses the equivalent and more current call:
    
    - [`ec2.connection.EC2Connection.run_instances`](http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances)
    
    This is a useful change in that connection.run_instances exposes
    additional features such as the capability of handling ebs_optimized
    instances, which image.run does not handle.
    
    This improvement was first suggested by @nchammas in a boto issue:
    https://github.com/boto/boto/issues/3054

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mrdrozdov/spark ad-spark-6474-useRunInstances

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/5172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5172
    
----
commit 6d14787fe05b93aa029eec6707aa294edb056897
Author: Andrew Drozdov <an...@mrdrozdov.com>
Date:   2015-03-24T18:03:21Z

    [SPARK-6474][EC2] Use EC2Connection.run_instances to launch clusters
    
    Instead of using the following call in spark_ec2.py to launch clusters:
    
    - [`ec2.image.Image.run`](http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.image.Image.run)
    
    this patch uses the equivalent and more current call:
    
    - [`ec2.connection.EC2Connection.run_instances`](http://boto.readthedocs.org/en/latest/ref/ec2.html#boto.ec2.connection.EC2Connection.run_instances)
    
    This is a useful change in that connection.run_instances exposes
    additional features such as the capability of handling ebs_optimized
    instances, which image.run does not handle.
    
    This improvement was first suggested by @nchammas in a boto issue:
    https://github.com/boto/boto/issues/3054

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by mrdrozdov <gi...@git.apache.org>.
Github user mrdrozdov commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85682470
  
    Congrats on your newfound power @nchammas :grinning: 



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5172#discussion_r27512141
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -502,8 +502,10 @@ def launch_cluster(conn, opts, cluster_name):
                                     if opts.additional_security_group in (sg.name, sg.id)]
         print "Launching instances..."
     
    +    # Check if the AMI exists
         try:
    -        image = conn.get_all_images(image_ids=[opts.ami])[0]
    +        if conn.get_image(image_id=opts.ami) is None:
    +            raise Exception()
    --- End diff --
    
    Do we want to include some reason here, or will this make it clear to the user what the error was? (Haven't tested this latest update out yet.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87749983
  
    Change looks fine to me but I haven't had a chance to run it yet.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87778725
  
      [Test build #29418 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29418/consoleFull) for   PR 5172 at commit [`4be463d`](https://github.com/apache/spark/commit/4be463d8468d971318a6e984fbe7cbb88b6da167).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on a diff in the pull request:

    https://github.com/apache/spark/pull/5172#discussion_r28174579
  
    --- Diff: ec2/spark_ec2.py ---
    @@ -502,8 +502,10 @@ def launch_cluster(conn, opts, cluster_name):
                                     if opts.additional_security_group in (sg.name, sg.id)]
         print "Launching instances..."
     
    +    # Check if the AMI exists
         try:
    -        image = conn.get_all_images(image_ids=[opts.ami])[0]
    +        if conn.get_image(image_id=opts.ami) is None:
    +            raise Exception()
    --- End diff --
    
    Does this bubble up a clear message to the user about what the problem is?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85707428
  
      [Test build #29106 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29106/consoleFull) for   PR 5172 at commit [`6d14787`](https://github.com/apache/spark/commit/6d14787fe05b93aa029eec6707aa294edb056897).
     * This patch **passes all tests**.
    
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85679274
  
      [Test build #29106 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29106/consoleFull) for   PR 5172 at commit [`6d14787`](https://github.com/apache/spark/commit/6d14787fe05b93aa029eec6707aa294edb056897).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by shivaram <gi...@git.apache.org>.
Github user shivaram commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87777699
  
    Jenkins, test this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85677434
  
    LGTM.
    
    @mrdrozdov Can you post a few invocations that you used to test things out? Change looks totally good and harmless, but it doesn't hurt to double check (especially since spark-ec2 has no tests apart from `pep8`). :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87802792
  
      [Test build #29418 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29418/consoleFull) for   PR 5172 at commit [`4be463d`](https://github.com/apache/spark/commit/4be463d8468d971318a6e984fbe7cbb88b6da167).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.
     * This patch does not change any dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85709569
  
    OK, tried those commands and everything checked out OK. The error when the AMI was bad was a bit verbose, but that's what we had before I guess.
    
    cc @JoshRosen @shivaram


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85682085
  
    Hohoho I guess things have changed around here. :sunglasses: 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by nchammas <gi...@git.apache.org>.
Github user nchammas commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85677806
  
    Jenkins, test this please. (Though I doubt Jenkins will listen to me.)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87802804
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29418/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85675496
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by mrdrozdov <gi...@git.apache.org>.
Github user mrdrozdov commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85690344
  
    Launching/Accessing/Destroy clusters using spark-ec2 should be sufficient for this change. Could run the following basic flow to double check that everything's in order:
    
    ```
    # Basic
    ./spark-ec2 -k <key_name> -i <key_path> -s 1 launch sparky
    ./spark-ec2 -k <key_name> -i <key_path> login sparky
    ./spark-ec2 -k <key_name> -i <key_path> destroy sparky
    
    # Bad AMI (HVM)
    ./spark-ec2 -k <key_name> -i <key_path> -s 1 -a ami-35b1885c launch sparky
    
    # Custom AMI
    ./spark-ec2 -k <key_name> -i <key_path> -s 1 --region=us-east-1 --zone=us-east-1d -a ami-5bb18832 launch sparky
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by mrdrozdov <gi...@git.apache.org>.
Github user mrdrozdov commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87768950
  
    Jenkins, test this please. (overly optimistic that this might work :stuck_out_tongue_winking_eye:)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-85707450
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29106/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-6474][EC2] Use EC2Connection.run_instan...

Posted by mrdrozdov <gi...@git.apache.org>.
Github user mrdrozdov commented on the pull request:

    https://github.com/apache/spark/pull/5172#issuecomment-87768741
  
    Great. I made a slight change to make reduce the verbosity when given a non-existent AMI. connection.get_image is definitely right call to use here, but there's a slight bug/inconsistency in boto. get_image is supposed to just return null if the AMI doesn't exist, but it ends up throwing an EC2ResponseError which I think was causing the verbosity that @nchammas noticed. I have a PR out for the boto fix, and in the mean time wrapped get_image in a try/except block.
    
    I could have just left the code as before, but I think it's nice to remove the image variable seeing that it's no longer being used, plus get_image is a bit more explicit than get_all_images. Although, it's n2bd.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org