[jira] [Created] (SPARK-3398) Have spark-ec2 intelligently wait for specific cluster states

["Nicholas Chammas (JIRA)" <ji...@apache.org>]

Nicholas Chammas created SPARK-3398:
---------------------------------------

             Summary: Have spark-ec2 intelligently wait for specific cluster states
                 Key: SPARK-3398
                 URL: https://issues.apache.org/jira/browse/SPARK-3398
             Project: Spark
          Issue Type: Improvement
          Components: EC2
            Reporter: Nicholas Chammas
            Priority: Minor


{{spark-ec2}} currently has retry logic for when it tries to install stuff on a cluster and for when it tries to destroy security groups. 

It would be better to have some logic that allows {{spark-ec2}} to explicitly wait for when all the nodes in a cluster it is working on have reached a specific state.

Examples:
* Wait for all nodes to be up
* Wait for all nodes to be up and accepting SSH connections (then start installing stuff)
* Wait for all nodes to be down
* Wait for all nodes to be terminated (then delete the security groups)

Having a function that blocks until the desired cluster state is reached would reduce the need for various retry logic and probably also eliminate the need for the {{--wait}} parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org